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ABSTRACT 

This report summarizes the National Job Training 
Partnership Act (JTPA) Study, which examined the impact of Title IIA 
of JTPA on participants' employment, earnings, and welfare receipt. 
Between 1987 and 1989, more than 20,000 adults and out-of-school 
youths who applied for JTPA services in 16 areas across the country 
were randomly assigned to a "treatment group" or to a "control" group 
that was ineligible for JTPA-funded services for 18 months but that 
remained eligible for all non-JTPA-f unded services in their 
communities. This study evaluation report covers the following 
topics: origins and goals of the national JTPA study; the study's 
setting and design; the JTPA program in the study sites; 
implementation of random assignment within the JTPA program; service 
receipt by treatment and control groups; sample size, data 
collection, and related issues; and site selection and the study's 
external validity. Profiles of the study sites are ap^^ended, 37 
exhibits are included. (Contains 23 references. (MN) 
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EXECUTIVE SUMMARY 



The Job Training Partnership Act (JTPA) is the nation's largest employment and training program 
for economically disadvantaged adults and youths. Its prominence — and the ongoing debate about its 
design, administration, and evaluation — makes the National JTPA Study an important and timely one, 
for it has produced the first analysis of JTPA's impacts in a diverse sample of sites. It was also the first 
effort to use a random assignment research design to estimate the impacts of local programs in an ongoing, 
voluntary, national employment and training program and included as well attempts to develop improved 
nonexperimental (non-random assignment) methods for estimating program impacts. Adults and out-of- 
school youths served under Title II-A, the largest component of the program, were the focus of the study. 

Pathbreaking in many ways, the project confronted serious issues of research design and 
implementation. This report, summarizing material from other project reports, provides background on 
how the U.S. Department of Labor — the federal agency that oversees JTPA — and the researchers 
balanced tradeoffs faced in the study and addressed threats to its validity. It pertains to the experimental 
portion of the study and is intended as a companion voiume to the first impact report, which provides 
estimates of program impacts based on 18 months of follow-up. (A subsequent report will present 30- 
month impact findings.) 

L The Origins and Goals of the Study 

The Job Training Partnership Act of 1982 established a nationwide network of more than 600 
federally funded, state-supervised, and locally operated partnerships between the public and private sectors 
to arrange for services intended to increase the employability of eligible adults and youths. Each year 
since its formal start-up in 1983, Title II-A (which was the main title of the legislation until 1993) 
received close to $2 billion and enrolled from 600,000 to one million individuals — representing less than 
10 percent of the JTPA-eligible population. The JTPA statute required the U.S. Department of Labor to 
determine whether the legislative goals of in'^reasing employment and earnings and reducing welfare 
receipt were met. 

In 1986, the Department initiated the National JTPA Study, a major effort to evaluate rigorously, 
in a limited number of local service delivery areas (SDAs) across the country, the impact that Title II-A 
of JTPA had on participants' employment, earnings, and welfare receipt. Following the recommendation 
of a technical Advisory Panel, the Department chose to conduct the study using a classical random 
assignment design. The Department outlined four major goals for the experimental portion of the study: 

1 
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• Select up to 20 local programs that represent the diversity of the national JTPA 
program. 

Estimate the overall impact of JTPA Title H-A activities provided in these local 
programs for both the fiiU sample and for important subgroups. 

• Estimate the specific impact of important categories of activities such as on-the- 
job training and classroom occupational training, again for the full sample and 
subgroups. 

• Conduct the analysis while changing the normal operations of the program as little 
as possible. 

Between 1987 and 1989, more than 20,000 adults and out-of-school youths who applied for JTPA 
in 16 local areas across the country were randomly assigned to a "treatment" group eligible to receive 
JTPA services or to a "control" group that was not eligible for JTPA-fiinded services for 18 months but 
remained eligible for all non-JTPA-funded services in the community. This study was designed to 
compare credibly and reliably how well the treatment group fared compared to the control group in 
meeting the program's goals - thereby showing the difference, or "impact," that JTPA services made over 
and above what would have happened to these individuals without the program. 

n. A Framework for Assessing Issues in the Study 

Properly implemented, random assignment is widely viewed as providing more accurate and 
reliable estimates of program impacts in the specific instance studied than using statistical matching to 
create a comparison group. As will become apparent in this report, howeve?, implementing a random 
assignment study in a group of sites that include the diversity of a complex system such as JTPA is a 
challenge. The choice of random assignment as a method for estimating program impacts was no panacea 
for eliminating all obstacles to success. 

In developing and implementing the study, the Department iiiid the researchers on the project faced 
many issues that posed threats to the success of the study. These threats can be grouped into those 
relating to the "internal validity" of the findings (i.e., the extent to which they are unbiased estimates of 
program impacts for the sites in the study) and to the "external validity" of the study findings (i.e., the 
extent to which they can be generalized to the larger population of JTPA sites). More specifically, this 
report addresses the following key issues confronted in this study: 

• Was random assignment implemented properly? Was random assignment 
placed at the proper point to capture the major program effects to be measured? 



Were normal program operations changed in important ways by introducing 
random assignment? Was the assignment process done in a random way, and do 
the observable characteristics of the control group and treatment group look the 
same? 

• Does the JTPA **treiiitinent** for which impacts are estimated have 
operational meaning? Was it possible to identify service strategies for which 
impacts could be estimated? What JTPA services did members of the treatment 
group receive overall and by service strategy? 

• What was the experience of the control group (the "counterfactuar* for the 
impact estimates), and how should the impact findings be interpreted? What 
was the actual service difference between the control group and the treatment 
group? How did this affect the interpretation of observed impacts? 

• What impacts could be detected with the available outcome data and sample 

size? How was information collected on key outcomes, and what issues arose in 
this effort? What size program impacts could be detected with the available 
sample for the 18-month follow-up period? 

• To what extent are the findings from the study externally valid, that is, 
generalizable to the larger JTPA system? What was the process for selecting 
study sites, and what does this imply about any special characteristics they may 
have? Do the environment in which the study sites operate, gross measures of 
their performance, and their administrative structure and services suggest that they 
are an unusual group of SDAs, or do they appear to include much of the diversity 
of the JTPA system? 

m* Implementing Random Assignment in the Study Sites 

The research design for the project had to recognize and accommodate the variety of local JTPA 
programs in the study sites while providing a means to address the central research questions posed by 
the LIS, Department of Labor. As a starting point, the researchers identifaed "generic" steps common to 
all the study sites, which potential JTPA enrollees typically followed during program intake. These 
generally involve: efforts by the SDA or service provider staff to recruit eligible and interested clients, 
application for the program and determination of eligibility, an assessment of client interests and current 
skills, development of a ser\'ice plan and recommendation of specific services, efforts to arrange the 
appropriate services, enrollment in JTPA, participation in program services, and termination from the 
program. It is common for individuals to "exit" from the intake process at each stage. Some find a job 
or other training on their own; others decide that JTPA is unlikely to provide what they want; and still 
others may be discouraged by what staff tell them about their prospects in the program. 



Three key features of this typical client flow should be emphasized. First, although JTPA funding 
is sufficient to serve less than 1 0 percent of the eligible population, many local programs in the late 1980s 
- a period of economic expansion - had to recruit applicants. Some study sites had to expand their 
recruitment efforts during the period of random assignment to meet their enrollment targets and 
accommodate the creation of a control group. Second, the typical intake process was designed to allow 
individualization of services; JTPA is not a common program model, even within a single SDA. 
Consequently, the research design had to accommodate many different types of service recommendations 
and estimate impacts for general service strategies rather than narrowly defined services. Third, the exit 
of applicants fipom the system at each step means that the placement of random assignment in the client 
flow would have affected who was in the sample and the percentage of those who were assigned to the 
treatment group and actually participated in program services. For example, if random assignment were 
placed early in the intake process, fewer people would have exited from the intake process, but a smaller 
percentage of those in the treatment group would actually participate in JTPA than if random assignment 
occurred later. 

Exhibit ES.l shows the random assignment design chosen for the impact study. Random 
assignment followed local staff determination of eligibility, assessment of interests and service needs, 
completion of an informed consent form and a Background Information Form (BIF), and recommendation 
of JTPA services and a service strategy. The three service strategies for which impacts were estimated 
were: (1) classroom training, the category for individuals recommended for classroom training in 
occupational skills and not for on-the-job training (OJT); (2) OJT/JSA Oob search assistance), the category 
for individuals recommended for OJT alone or in combination with job search assistance and not for 
classroom training in occupational skills; and (3) other services, for all other cases. 

For each service strategy, a treatment and a control group were created through a random process, 
which produced treatment and control groups with similar pre-random assignment characteristics, based 
on the information provided on the BIF. In general, this random process produced two treatment group 
members for each control group member. Thus, one of the most serious potential threats to the validity 
of the study noncomparable groups at the beginning of the follow-up period — was not a problem. 

If an individual was randomly assigned to the treatment group, local staff worked to arrange 
services consistent with his or her pre-random assignment service strategy recommendation. Individuals 
randomly assigned to the control group were unable to receive JTPA services for a period of 1 8 months 
following random assignment, but remained eligible for other services in the community and were 
provided with a list of these alternatives. 



Local program staff did follow these rules carefully, and fewer than 3 percent of the control group 
were enrolled in JTPA at any point in the 18-month follow-up period, thus averting a second serious 
potential problem. As discussed below, however, some members of the control group sought and found 
alternative services, and this must be taken into account in interpreting the impact findings. 

This research design addressed the two central impact questions: 

JTPA service strategy impacts were estimated by comparing the experiences of 
the treatment group for each strategy with those of the control group for that 
strategy. 

• Overall JTPA program impacts were estimated by combining the three 
treatment groups (recommended for the classroom training, OJT/JSA, and other 
services strategies) and comparing their experience follow ing random assignment 
with a combined control group made up of the three separate control groups. 

The first type of estimate measures the impacts of the three service strategies for those people local 
staff recommended for each. Thus, policymakers can assess the impacts of classroom-based services 
(primarily training in occupational skills, but also basic education), services with an emphasis on 
immediate employment with or without training (on-the-job training and job search), and a mixture of 
other services. Hc^wever, this design did not produce experimental comparisons of the relative (or 
"differential") impacts of service strategies because assignment to each of the three treatment groups and 
the control group was not random. It was expected, and was the case over the implementation of the 
scudy, that the groups recommended by SDA staff for the three service strategies would differ on many 
baseline characteristics. To produce an experimental estimate of the differential impact of service 
strategies, individuals would have had to be randomly assigned to these service strategies to assure a 
similarity in baseline characteristics. The choice of research design rested on an implicit judgment by the 
Department and the researchers that testing JTPA as closely as possible to the way it normally operates 
was more important than producing experimental estimates of the differential impacts of the service 
strategies - that is, comparisons of the impacts of service strategies for similar groups. 

The second type of estimate allows an assessment of how the overall JTPA program in the study 
sites benefited different types of individuals, but here the interpretation of the impact estimates is difficult. 
The research plan produced service differences between experimentals and controls within the three 
individual service strategies (for example, in the classroom training service strategy, the treatment group 
received more classroom training than the controls and in the OJT/JSA strategy they received more OJT 
than the controls), but when all three service strategies are combined it is difficult to describe and interpret 




the nature of the service differential. Thus, as much as possible, we seek to present impacts for the 
specific service strategies. 

Although efforts were made to change the operation of local JTPA programs as little as possible, 
some changes inevitably did occur. First, some of the study sites had to expand outreach efforts to recruit 
a sufHcient applicant pool to serve the planned number of clients and to allow creation of a control group. 
This, in combination with the gradually declining unemployment rate in many of the sites, led them to 
recruit and serve a somewhat less job-ready group. Second, in some sites, the selection of a service 
strategy for each individual prior to random assignment may have prevented stafT, in unusual cases, from 
changing their minds about the proper service recommendation after a person was randomly assigned to 
the treatment group. 

Overall, however, the study did not appear to affect the nature of the services provided to members 
of the treatment group following assessment and random assignment. Staff continued their normal 
practices in referring them either to in-house services provided by SDA staff (most often job search 
assistance or on-the-job training) or to outside service providers. 

One final issue in the implementation of random assignment concerns whether the measured pre- 
random assignment characteristics of the treatment and control groups were similar. The Background 
Information Form completed prior to random assignment collected many baseline characteristics on sample 
members. A multivariate discriminant analysis found that there was virtually no correlation between 
whether a person was a member of the treatment or control group and his or her measured baseline 
characteristics. Any differences in baseline characteristics between the two groups were neither substantial 
nor statistically significant. 

rV, The Nature of the Experimental Service Comparisons 

The experimental impact estimates presented in the National JTPA Study are based on a 
comparison of the experiences of two similar groups, both of which were found eligible and appropriate 
for JTPA services. As such, the estimated impacts are for the added post-assessment services made 
possible by access to JTPA as compared to a "background" level of other services — including JTPA 
assessment and other (non-JTPA) services — in which control group members participated. 

Two points are vital for understanding the nature of the impact estimates. First, the treatment 
group did not all receive the JTPA services for which they were recommended, for reasons discussed 
below. Sixty-four percent of the treatment group were enrolled in JTPA at some point in the follow-up 
period, and the services they received were generally consistent with the service strategy designations 
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made by local staff prior to random assignment. Of the remaining 36 percent of the treatment group, 
about half received some type of JTPA assistance after random assignment that did not result in an 
enrollment (and generally was not intensive), while the other half received no significant post-random 
assignment service. Second, the impact estimates reported in the study are not for a comparison of JTPA 
services versus no services because some members of the control group sought and found alternatives to 
JTPA. 

The three service strategies used in the impact analysis did represent distinct JTPA service 
emphases, as shown in Exhibit ES.2. The exhibit presents the key JTPA services in each strategy, using 
two different bases for the comparison (the entire treatment group and those in the treatment group who 
were enrolled in JTPA): 

• Classroom trainiag. Most enroUees were active in classroom-based services of 
some type, with classroom training in occupational skills (CT-OS) being more 
prevalent than basic education (BE). Eighty-six percent of adult male enrollees, 
89 percent of adult female enrollees, 80 percent of male youth enrollees, and 86 
percent of female youth enrollees were active in classroom training in 
occupational skills, basic education, or both. 

OJT/JSA. Most enrollees were active in on-the-job training and/or job search 
assistance. Eighty-seven percent of adult male enrollees, 88 percent of adult 
female enrollees, 85 percent of male youth enrollees, and 85 percent of female 
youth enrollees were active in either or both activities. 

Other services. The predominant types of services differed between adults and 
youths. Eighty-nine percent of adult male enrollees and 82 percent of adult 
female enrollees were active in job search assistance or miscellaneous services, 
while 83 percent of male youth enrollees and 80 percent of female youth enrollees 
were active in basic education or miscellaneous services. 

A follow-up survey conducted approximately 18 months after random assignment provided 
information on many non-JTPA employment and training services in which members of the treatment and 
control groups participated. It was possible to collect reasonably complete survey data on the receipt of 
classroom training in occupational skills and basic education for both groups. Respondents to the survey 
could not or did not distinguish on-the-job training and paid work experience from regular employment, 
so no information is available on participation in these activities outside JTPA. However, JTPA 
participation in these activities by the treatment and control groups (as measured fix)m SDA records) is 
probably a reasonable indicator of the treatment-control service difference because JTPA is the primary 
source of these activities. Finally, SDA records showed participation in JTPA-provided job search 
assistance and miscellaneous services. However, the survey did not provide information on non-JTPA 
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EXHIBIT ES^2 



KEY SERVICES FOR TREATMENT GROUP MEMBERS AND ENROLLEES, 
BY TARGET GROUP AND SERVICE STRATEGY 



Sample and 
Target Group 



Classroom 
Training 



OJT/JSA 



Other 
Services 



Treatment 

Group Members 



Adult Men 



61% enrolled in 
CT-OS or BE 



49% enrolled in 
OJT or JSA 



52% enrolled in 

JSA or misc. services (a) 



Adult Women 



65% enrolled in 
CT-OS or BE 



49% enrolled in 
OJT or JSA 



52% enrolled in 

JSA or misc. services (a) 



Male Youths 



60% enrolled in 
CT-OS or BE 



49% enrolled in 
OJT or JSA 



56% enrolled in 

3E or misc. services (a) 



Female Youths 



62% enrolled in 
CT-OS or BE 



49% enrolled in 
OJT or JSA 



50% enrolled in 

BE or misc. services (a) 



Eniollecs 
Adult Men 

Adult Women 

Male Youths 

Female Youths 



86% enrolled in 
CT-OS or BE 

89% enrolled in 
CT-OS or BE 

80% enrolled in 
CT-OS or BE 

86% enrolled in 
CT-OS or BE 



87% enrolled in 
OJT or JSA 

88% enrolled in 
OJT or JSA 

85% enrolled in 
OJT or JSA 

85 % enrolled in 
OJT or JSA 



89% enrolled in 

JSA or misc. services (a) 

82% enrolled in 

JSA or misc. services (a) 

83% enrolled in 

BE or misc. services (a) 

80% enrolled in 

BE or misc. services (a) 



SOURCE: MDRC calculations from Background Information Form responses and from 
program enrollment and participation data from the 16 SDAs. 

NOTES: Calculations for this table are based on data for all treatment group members and 
enroUees in the 18-month study sample. 

(a) Miscellaneous program services include such activities as assessment, job-readiness 
preparation, vocational exploration, job shadowing, and tiyout employment. In this Uble, 
miscellaneous program services also include work experience, which is identified as 
a separate program service in other reports on the National JTPA Study. 
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sources of these activities, so it is not possible to report a service difference for these activities. It is 
likely, though, that many members of the control group received some assistance from state Employment 
Services, primarily in the form of job leads. 

From the available information on JTPA and non-JTPA service receipt, it is possible to characterize 
the service difference between the treatment and control groups. Overall, there was a difference, but it 
was modest in size. Despite the data problems discussed above, clear patterns of service differences for 
individual service strategies and target groups (male and female adults and male and female youths) do 
appear in the data: 

For the classroom training service strategy, clear treatment-conrrol group 
differences in the key service - classroom training in occupational skills - did 
emerge for all target groups. For basic education, the treatment and control group 
average hours were very similar for adults and male youths, while female youths 
in the treatment group averaged somewhat more hours in this service than did 
their control group counterparts. 

For the OJT/JSA service strategy, using the assumption that JTPA is the 
primary source of on-the-job training, the treatment group averaged substantially 
more hours of this service than did the control group for each target group. 
Because of the problems of measuring non-JTPA participation in job search 
assistance discussed above, it was not possible to calculate an hours difference for 
this service. 

• For the other services strategy, the problems encountered in measuring non- 
JTPA program participation in job search assistance and miscellaneous services 
(which are the most common services for adults) prevented calculation of service 
differences for these two activities. For youths, basic education replaces job 
search assistance as an important service. Interestingly, for female youths, there 
was virtually no difference in hours in basic education between the treatment 
group and controls, while for male youths, the control group actually participated 
more. 

The impact estimates for the overall sample and for the three service strategies presented in the 
accompanying IS-month impact report must be interpreted as measuring the effect of these modest 
differences in services between the treatment and control groups. For example, small positive impacts 
would be more encouraging in light of the modest service difference than would be the case had the 
service difference been much larger. 

The size of the available sample for impact estimates also affects the ability to detect the impact 
of the services under study. The overall experimental sample of 20,601 is large, as is the 18-month 
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impact sample of 14,441 individuals. The difference between the total number randomly assigned and 
the smaller impact sample for the 18-month impact report arises because of the exclusion of three groups: 

• 473 "extra" treatment group members who had been randomly assigned at a ratio 
higher than 2 to I (to ease the recruitment burden on sites) were excluded from 
the analysis so that impact estimates would not involve complex weighing 
procedures. 

• 3,097 people scheduled for follow-up interviews prior to 18 months after random 
assignment were excluded to avoid the complexities introduced by variable-length 
follow-up. 

• 2,584 people (15 percent of the attempted surveys) were excluded because they 
did not respond to the 1 8-month survey. 

A sample of more than 14,000 people is clearly adequate for estimating the impacts of the overall 
JTPA program for the full sample. The real question concerns how far this sample can be disaggregated 
into the four target groups, the service strategies, and further subgroups. Not surprisingly, despite this 
large sample, it is not possible to estimate impacts with precision for some combinations of target groups 
and service strategies. 

V. Drawing Conclusions About the National JTPA System from Study Findings 

An important issue is the extent to which findings from the sites in the study can be generalized 
to the JTPA system as a whole. In a strict statistical sense, study findings are externally valid when sites 
are randomly chosen and, hence, representative of what would have been found had all possible programs 
been studied. Prior to this study (to the best of the knowledge of the researchers working on the project), 
no comparable social experiment had undertaken such an approach. In light of its advantages, however, 
the Department and the researchers attempted to select a random sample of local JTPA programs. For 
a variety of reasons, the percentage of randomly selected sites that agreed to participate was low, making 
a "probabilistic" selection of sites infeasible. Instead, the researchers contacted local programs with a goal 
of recruiting sites that collectively reflected the diversity of the JTPA system rather than a narrow 
subsample of it. 

Several criteria can serve as practical guidelines to help readers assess the extent to which the 
findings from these sites should be used to draw broader conclusions about the JTPA system: 

• Process of selection. Was the process used to recruit and select sites designed 
or implemented in a way likely to recruit sites that would produce unusually 
strong or weak findings? 



« Geographic and **environmentar* diversity. Do the sites include the diversity 
of the overall "universe" of sites? 

♦ Operational diversity. Do the sites include the diversity of administrative 
structure, program emphasis, client characteristics, and richness of funding and 
services seen in the national program? 

« Operational maturity and stability* Are the sites established programs rather 
than new entities, recently reformed programs, or agencies facing unusual 
administrative stress? 

While the process of selection was clearly not random, the researchers contacted potential sites 
based on their region, size, labor market conditions, and other similar characteristics rather than the 
strength of their JTPA programs. Obviously, no information was available on local program impacts, so 
this could not be used to guide the selection process. JTPA standards (based on the success of SDAs in 
placing participants in jobs or achieving other favorable outcomes) have routinely been used to assess the 
success of local programs; the experience of local programs in meeting or exceeding these standards was 
known at the time of site selection. However, these standards focus on program outcomes rather than 
impacts; the sites chosen show a distribution of performance similar to the national JTPA system; and - 
when the first site-level impacts were estimated - there proved to be no relationship between success in 
meeting key standards and site impacts. 

The sites in the study, shown in Exhibit ES.3, as a group do resemble the national JTPA system 
in many ways, including labor market conditions and JTPA program performance. Reflecting the variety 
of JTPA nationally, the study sites include SDAs with varying administrative structure, types of service 
providers, degree of centralization of intake and service provision, and service emphasis. Additionally, 
the sample of individuals for which impacts are estimated is similar in age, ethnicity, and work experience 
to those persons served nationally by JTPA. 

There are three notable exceptions to this general picture of similarity of the study sites to the 
national system. First, there is no large, central city among the sites, although the study does include two 
SDAs (Oakland and Jersey City) that share many of the labor market and demographic features of such 
areas. Second, no SDAs serving very small numbers of people were included in the study because of the 
large fixed costs of conducting the research in a site and the small contribution they would have made to 
the sample. Finally, the study sites provided less on-the-job training and miscellaneous services, and more 
classroom training in occupational skills and job search assistance, than was the case nationally at the time 
of the study. 
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EXHIBIT ES.3 

KEY FACTS ABOUT THE NATIONAL JTPA STUDY SITES 



Site Name 


SDA Name 


Census 
Region 


Largest City 


18-Month 

Study 
Sample Size 


Butte, Mont. 


Concentrated Employment 


West 


Butte 


477 




Program, Mont. 








Cedar Rapids, Iowa 


East Central Iowa 


Midwest 


Cedar Rapids 


346 


Coosa Valley, Ga. 


Coosa Valley, Ga. 


South 


Rome 


1,806 


Corpus Christi, Tex. 


Corpus Christi/Nueces 


South 


Corpus Christi 


1,498 




County, Tex. 








Decatur, III. 


Macon/De Witt 


Midwest 


Decatur 


471 




Counties, 111. 








Fort Wayne, Ind. 


Northeast Indiana 


Midwest 


Fort Wayne 


2,559 


Heartland, Fla. 


Heartland, Fla. 


South 


Lakeland 


597 


Jackson, Miss. 


Capital Area, Miss. 


South 


Jackson 


1,375 


Jersey City, N.J. 


Corporation for Employ- 


Northeast 


Jersey City 


1,170 




ment and Training, Inc. 








Larimer County, Colo. 


Larimer County, Colo. 


West 


Fort Collins 


668 


Marion, Ohio 


Crawford/Hancock/Marion/ 


Midwest 


Marion 


1,083 




Wyandot Counties, Ohio 








Northwest Minnesota 


Northwest Minnesota 


Midwest 


Thief Ri\cjr 


498 




(Crookston and Thief 




FaUs 






River Falls) 








Oakland, Calif. 


Oakland, Calif. 


West 


Oakland 


1,048 


Omaha, Neb. 


Greater Omaha, Neb. 


Midwest 


Omaha 


956 


Providence, R.I. 


Providence/Cranston, R.I. 


Northeast 


Providence 


1,277 


Springfield, Mo. 


Job Council of the 


Midwest 


Springfield 


1,202 




Ozarks, Mo. 








All Sites 








17,031 
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Because of the site selection process, the results from this study should not automatically be 
generalized to the national JTPA system as a whole. Instead, they should be seen as implementation and 
impact findings for a group of local programs that reflect much, but not all. of the diversity of JTPA. The 
analysis of program implementation and impacts contained in other reports from the project seeks to 
determine how client characteristics, local circumstances, and program design decisions affect key findings 
among the sites, allowing the reader to assess more fiilly the applicability of the findings to the larger 
JTPA system. 



The National JTPA Study was not the first time random assignment had been used to assess the 
impact of employment and training programs. Its successfiil use elsewhere was one of the reasons the 
U.S. Department of Labor chose this apprcich for the present project. However, the ambitious goals of 
the project, the decentralized and diverse nature of the ongoing local JTPA programs, and the desire to 
study the program while changing normal operations as little as possible combined to create a major 
challenge for the research team. Most, but not all, of the original objectives of the study have been 
achieved. In the course of implementing the study, much has been learned about the operation of JTPA: 
how SDAs stmcture themselves administratively to operate the program, recruit clients, and recommend 
and arrange services. The impact analysis, as intended, has provided estimates of the overall impact of 
post-assessment JTPA services and the impact of key service strategics. These estimates are available for 
four target groups and other key subgroups. Unfortunately, the unprecedented effort to recniit a 
probabilistically selected group of local SDAs was not successful, but the final study sites do include much 
of the diversity of the national JTPA system. 

In developing and implementing the research design for this project, the Department and the 
research team confronted many issues that forced them to assess the relative importance of the many 
competing goals of the study. This report has summarized some of these key decision points, outlined 
the tradeoffs involved, and explained the choices made as the project moved ahead. With this background, 
readers can better understand the implementation and impact findings presented in the other project reports 
and assess for themselves the wisdom of the final research design. 
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CHAPTER 1 

THE ORIGINS AND GOALS OF THE NATIONAL JTPA STUDY 



For a decade, the Job Training Partnership Act of 1982 (JTPA) has been the nation's largest and 
most visible program providing employment and job training opportunities for economically disadvantaged 
individuals. JTPA established a nationwide network of more than 600 federally funded, state-supervised, 
and locally operated partnerships between the public and private sectors to arrange for services intended 
to increase the employability of eligible adults and youths. Each year since its formal start-up in 1983, 
Title II-A (the main title of the legislation) has received close to $2 billion and enrolled between 600,000 
and 1 million individuals - which represents less than 10 percent of the JTPA-eligible population. 

The JTPA statute required the U.S. Department of Labor (hereinafter referred to as "the 
Department"), the federal agency that oversees JTPA, to conduct an evaluation to determine whether the 
legislative goals of increasing employment and earnings and reducing welfare receipt were met. In 1986, 
the Department initiated the National JTPA Study, a major effort to evaluate rigorously, in a limited 
number of the local service delivery areas (SDAs)' across the country, the impact that Title II-A of JTPA 
had on participants* employment, earnings, and welfare receipt.^ Following the recommendation of a 
technical Advisory Panel, the Department adopted and oversaw the implementation of a social experiment 
using a classical random assignment design. The Manpower Demonstration Research Corporation 
(MDRC) was chosen as the prime contractor (with Abt Associates Inc. as a subcontractor) to assist the 
Department in selecting the sites, implement the study design in the sites, and describe the processes by 
which the study was implemented and the programs were operated. Abt Associates Inc. was chosen as 
the prime contractor (with ICF, MDRC, New York University, and the National Opinion Research Center 
as subcontractors) to develop the research design, collect baseline and follow-up data, and conduct the 
analysis of the program's impacts and of its benefits compared to its costs. 

Between 1987 and 1989, more than 20,000 adults and out-of-school youths in 16 local areas across 
the country were randomly assigned to a "treatment" group eligible to receive JTPA services or to a 
"control" group that was not eligible for JTPA-funded services for 18 months but remained eligible for 



JTPA parlance, "service delivery area" refers to both the geographical area being served and the JTPA 
administrative unit serving it. 

^Thc only major Title II-A group not included in the study was in-school youths. This group was excluded 
because of sample size concerns and because the outcomes measured in the study (employment, earnings, and welfare 
receipt) would not have been relevant for this group during much of the study's 30-month follow-up period. 
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all non-JTPA-funded services in the community. This study was designed to compare credibly and 
reliably how well the treatment group fared compared with the control group in meeting the program's 
goals - thereby showing the difference, or "impact," that JTPA services made over and above what would 
have happened to these individuals without the program. This was one of the early examples of applying 
experimental techniques to evaluate an existing, ongoing service delivery system. 

The research design for the National JTPA Study had four main, interrelated parts. The first three 
focused on the 16 SDAs that participated in the evaluation: a study of their program operations and the 
way the study was implemented, a study of the program impacts, and a study of benefits generated by the 
program compared with its costs. The fourth part of the research design involv^^ap^ttempt to develop 
new nonexperimental methods to estimate program impacts. 

This report summarizes material contauicd in other project reports with the goal of helping the 
research and policy community understand the issues that arose in designing and implementing the 
"experimental" portion of the National JTPA Study^ and assess for themselves the ways in which they 
were addressed.* It brings together in one place material that will help the reader interpret the impact 
findings, which are presented in other volumes.^ The presentation presumes that the reader has some 
background in the issues surrounding program impact - and especially random assignment - research. 
The research reports cited throughout this document provide fiirther information on these subjects. 

This first chapter presents a brief overview of the origins and goals of this study, followed by a 
discussion of the basic issues confix)nted in conducting the study and the various "threats" to its validity 
that had to be addressed. These threats are grouped into two categories: those relating to the "internal 
validity" of the study findings (i.e., the extent to which they are unbiased estimates of program impacts 
for the sites in the study) and those bearing on the "external validity" of the study findings (i.e., the extent 
to which the findings can be generalized to the larger population of JTPA sites^). 

Chapters 2 through 7 of the report discuss how the Department and the researchers on the project 
addressed these issues. Chapter 2 begins this discussion by presenting an overview of the JTPA system. 
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^The nonexperimental analysis is discussed in Bloom et al., 1990. 

*Thc other reports in the project arc Bloom ct al., 1990; Doolittlc and Tracgcr. 1990; Bloom, 1991; Kcmple, 
Doolittlc, and Wallace, 1993; and Bloom ct al., 1993. They are listed at the front of this report. 

'Impact findings based on 18 months of follow-up are presented in Bloom ct ai., 1993, The final report, 
presenting impact findings based on 30 months of follow-up, will be released in late 1993. 

'To the extent that the results of a study can be generalized to different subjects and settings, the study possesses 
external validity. See Campbell and Stanley, 1963. 

1-2 



^ 23 



the study sites, and the research design. Tiie subsequent chapters focus on specific issues concerning the 
design and implementation of the research: 



Chapter 3: on the nature of the administrative system and services in the study 
sites. 

Chapter 4: on the implementation of random assignment, including changes made 
in normal JTPA practices as a result of implementing study procedures, and the 
comparability of the treatment and control group. 

Chapter 5: on the difference in employment and training services received by the 
treatment and control groups, including how the size of this differential should 
affect the interpretation of impact fmdings. 

Chapter 6: on sample size and data source issues, including the response rate for 
the follow-up survey, sample sizes and minimum detectable effects, and survey 
response bias and corrections. 

Chapter 7: on the process of selecting sites for the study and the sites' similarities 
to and differences from the JTPA system as a whole. 



L The Origins of the Study 

Section 454 of the Job Training Partnership Act directs the Department of Labor to study the 
effectiveness of programs authorized under JTPA in achieving three goals: increasing participants' 
employment and earnings, reducing income support costs, and increasing tax revenues. This statutory 
language calls for an analysis of program impacts, not post-program outcomes — a crucial distinction for 
understanding the study's research goals and approach.^ 

Program outcomes include such measures as post-program employment rates, earnings, and rates 
of receipt of welfare for people who participate in JTPA. These measures, which may be short term or 
long term and are used in the existing JTPA performance standards system, are not measures of the 
difference JTPA makes (that is, the program's impacts) because some of those served would have found 
a new job on their own or improved their skills and raised their income through other means even if they 
had not participated in the program. Thus, crediting all post-program "success stories" to JTPA 
overestimates the program's impacts. To estimate program impacts, there must be a way to measure what 
would have happened had individuals not been given access to the program under study. Subtracting this 
from the outcomes yields the impact measures. 



^In some evaluation literature, outcomes are referred to as gross effects and impacts as net effects. 
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The Department's choice of random assignment as the method to assess program impacts came after 
an examination of ahemative research methods to determine program impacts: 

Nonexperimental approaches that identify a matched comparison** group 
simihir to program participants* 

Members of the comparison group are individuals who appear similar to participants on measurable 
characteristics such as age, education, sex, race, work history, and previous earnings. Possible members 
of this comparison group are typically identified in data collected on members of the general public, most 
of whom have never applied to an employment and training program. Econometric techniques are used 
to "adjust" the findings for differences that do exist between program participants and tiie comparison 
group, allowing the behavior of participants and the comparison group to be compared over time.* 

• Use of classical experimental methods involving random assignment of 
program applicants to a **program" or "treatment" group given access to the 
program or to a "control" group not given access to the program. 

Random assignment is a lottery-like process in which large numbers of applicants for a program 
are assigned by a random process to a "treatment" group, which is given access to the program, or a 
"control" group, which is not. Because the two groups are created in a random way, there are no 
systematic differences between the prior experience and characteristics of the two groups at the time the 
study begins. Differences that appear over time in employment, earnings, and other outcomes can be 
attributed to the program itself. 

Both approaches had been used in past studies of employment and training programs.^ A brief 
historical account illustrates the issues that arise in choosing between them. 



n. Past Research on Employment and Traini ng Programs 

During the 1960s and 1970s, the Department of Labor invested billions of dollars in programs for 
adults and youth and funded many nonexperimental evaluations of their effects.^" The evaluation of the 
CETA program was among the largest undertaken during the late 1970s and early 1980s. The Department 



•See Bamow, 1987, for a review of techniques of this type that had been tried by the time the study bcg^. 

^Many cvaluaiic'^s of the Comprehensive Employment and Training Act (CETA) program relied on a comparison 
group methodology to assess program impacts, while random assignment was used to study several important new 
program models such a; Supported Work and JOBSTART, a variety of employment programs for dislocated workers 
under Title III of JTPA, and small- and large-scale employment programs for recipients of Aid to Families with 
Dependent Children (AFDC), the major federally funded welfare program. However, the JTPA study did differ in 
examining a large, ongoing, voluntary program in numerous sites. 

*^For a review of early research on employment and training programs, see Perry et al., 1975. For a review of 
research on youth programs, see Betsey et al., 1985. For a review of the research on the CETA program, see 
Bamow, 1987. 



M 30 



funded an initial analysis of program impacts'' and then supported reanalyses of the same data by other 
researchers (using slightly different techniques to identify a comparison group and analyze impacts). 

Unfortunately, tFiis series of studies, using state-of-the-art nonexperimental research techniques, did 
not produce a consensus on CETA's impactsJ^ The r.iost extreme examples of inconsistent results 
concerned adult men: Estimates of the overall CETA program's impact on annual earnings ranged from 
$200 to minus $700, and estimates of the impact of CETA's on-the-job training for white men ranged 
from a high of $1,000 to $1,200 to a low of minus $200. Estimates for other activities and subgroups 
had narrower ranges but still showed so much uncertainty that they were difficult to use for policy 
planning. Therefore, policymakers did not know if CETA actually increased participants' employment 
and earnings. 

The Department originally planned to study JTPA using a comparison group approach similar to 
that used for CITA.'^ As the Department was doing the fmal planning for this research, the series of 
inconsistent nonexperimental findings on CETA, discussed above, began to emerge. Seeking a consensus 
in the research community on the best way to conduct nonexperimental research on employment and 
training programs, in late 19S4 the Department appointed a panel of evaluation experts to review its plans 
for studying JTPA and to recommend ways in which to deal with the unresolved methodological problems. 
The panel devoted most of its attention to the central methodological problem of "selection bias." 

Possible selection bias has continually been an issue in nonexperimental research such as that done 
on CETA because of the difficulty of choosing the proper comparison group. Researchers can only match 
members of the participant group with other people on observed or measured characteristics such as 
income, education, sex, and ethnicity. If the probability of applying for and being accepted into a program 
were affected only by the type of observed characteristics that can be collected for large numbers of 
people, then selection into programs could be modeled statistically and could be taken into account in 
choosing a comparison group. But the probability of applying to and being selected for a program is also 
likely to be influenced by program staffs reaction to the personal ^pearance and demeanor, motivation, 
or work attitudes of applicants, which are very difficult to measure reliably and are not included in most 



''Wcstat, 1981. 

'^See Bamow, 1987, for a review of these studies and a discussion of the differences in methodology, coverage, 
and results. 

*^For a detailed discussion of the Department's original research plans for JTPA and a comparison with the 
earlier research on CETA, see Westat, September 1984. In fact, the originally proposed JTPA analysis included 
several advances over the methods previously used for studying CETA. The original JTPA study, for example, 
would have included a specially identified survey comparison group, rather than relying on a group identified in 
existing survey samples. 
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large data sources. In estimating program impacts, selection bias occurs if unobseired characteristics (not 
included in the statistical analysis modeling the selection process) affect the likelihood of participation in 
a program and also affect later experiences. 

After reviewing the CETA research record and the statistical and econometric techniques available 
to address the selection bias problem, the panel concluded: 

that the estimates of the net impact of CETA are not reliable and that the true impacts of 
CETA are still open to question. Since the methods intended to be used to evaluate the Job 
Training Partnership Act are broadly the same as those used to evaluate CETA, there is a 
considerable likelihood that the validity of the net impact evaluation of JTPA will also be 
subject to unresolvable doubt if the existing analysis design remains unchanged.'* 

The panel also found it very unlikely that any method of addressing the problem of selection bias would 
be found in time to conduct a valid impact study of JTPA." 

In considering the alternative of random assignment, panel members recognized a tradeoff. In their 
view, random assignment research, properly conducted, provides the best opportunity to obtain accurate 
impact estimates for the local programs included in the study (i.e., it maximizes the "internal validity" of 
the estimates). But the challenge of implementing random assignment in an ongoing program such as 
JTPA means that some sites that have been selected for the study will not want or be able to participate 
and that the total number of study sites will be less than in the planned comparison group analysis of 
program impacts. Thus, it would be difficult to include a representative group of programs to allow 
generalization to the entire JTPA program (i.e., to establish the "external validity" of the estimates). 

Balancing these factors, the panel felt that without internal validity little could be learned about 
program impacts. It therefore recommended that the Department abandon the planned comparison group 
research and instead conduct a series of classical experimental studies of the impacts of JTPA for selected 
target groups and activities in a limited number of SDAs that volunteered to participate.'* 

The National Academy of Sciences, in a separate review of research on employment and training 
programs for youth, reached a similar conclusion in 1985." Finding that comparison groups in past 
research often differed markedly from participant groups, the academy's Committee on Youth Employment 
Programs concluded: 



'*Job Training Longitudinal Survey (JTLS) Research Advisory Panel, 1985, p. 2. 
"JTLS Research Advisory Panel, 1985, p. 21. 
'*JTLS Research Advisory Panel, 1985, pp. 21-26. 
''Betsey et al., 1985. 
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that control groups created by random assignment yield research findings about employment 
and training programs that are far less biased than results based on any other method. . . . 
Future advances in field research on the efficacy of employment and training programs will 
require a more conscious commitment to research strategies using random assignment.^' 

After reviewing these recommendations, the Department decided to suspend the planned comparison 
group research and proceed with a set of classical field experiments. In mid- 1986, the Department 
announced that it would conduct a random assignment field study of JTPA to include up to 20 local 
service delivery areas. It also announced a parallel effort to improve nonexperimental techniques. 



in. The Research Goals of the JTPA Experimental Analysis and Threats to the Success of the 
Study 

In the Request for Proposals (RFP) issued in early 1986 to conduct a random assignment study of 
Title II-A of JTPA, the Department identified four main research goals for the experimental portion of 
the study: 

• Select up to 20 local programs that represent the diversity of the national 
JTPA program. 

• Estimate the overall impact of JTPA Title II-A activities provided in these 
local programs for both the full sample and for important subgroups. 

• Estimate the specific impact of important categories of activities such as on- 
the-job training and classroom occupational training, again for the full 
sample and subgroups. 

• Conduct the analysis while changing the normal operations of the program 
as little as possible. 

Property implemented, random assignment is widely viewed as providing more accurate and reliable 
estimates of program impacts than using statistical matching to create a comparison group.^^ As will 
become apparent in this report, however, implementing a random assignment study in a group of sites that 



'•Betsey et al., 1985, pp. 18, 30. 

^'A special survey would be conducted of individuals who were eligible for but did not apply to JTPA in some 
of the SDAs participating in the National JTPA Study. Data from this survey would be used in new efforts to 
develop a method to construct a comparison group similar to the control group created through random assignment. 
Supporting this effort would be an analysis of the process by which individuals are recruited and selected for JTPA 
participation in SDAs in the study. 

^^In the terminology of the evaluation literature, when properly implemented a random assignment design is 
generally viewed as more likely to produce "internally valid** estimates of program impacts: that is, accurate or 
"unbiased" answers to the question of whether the treatments did make a diflfercnce in the specific instances studied. 
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includes the diversity of a complex system such as JTPA is a challenge. Chapter 2 presents an overview 
of the research design developed for the study. 

In developing and implementing the study, the Department and the researchers on the project faced 
many issues that posed threats to the success of the study, including the following key ones: 

• Was random assignment implemented properly? Was random assignment 
placed at the proper point to capture the major program effects to be measured? 
Were normal program operations changed in important ways by introducing 
random assignmprtt? Was the assignment process done in a random way, and do 
the observah^c^aracteristics of the control group and treatment group look the 
same? ^ 

• Does the JTPA "treatment" for which impacts are estimated have operational 
meaning? What were the administrative procedures and service offerings of the 
study sites? Was it possible to identify service strategies for which impacts could 
be estimated? What JTPA services did members of the treatment group receive 
overall and by service strategy? 

• What was the experience of the control group (the "counterfactual" for the 
impact estimates), and how should the impact findings be interpreted? What 
was the actual service difference between the control group and the treatment 
group? How does this affect the interpretation of observed impacts? 

What impacts could be detected with the available outcome data and sample 
size? How was information collected on key outcomes, and what issues arose in 
this effort? What size program impacts can be detected with the available sample 
for the 18-month follow-up period? 

• To what extent are the findings from the study externally valid, that is, 
generalizable to the larger JTPA system? What was the process for selecting 
study sites, and what does this imply about any special characteristics they may 
have? Do the environment in which the study sites operate, gross measures of 
their performance, and their administrative structure and services suggest that they 
are an unusual group of SDAs, or do they appear to include much of the diversity 
of the JTPA system? 
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CHAPTER! 

AN OVERVIEW OF THE NATIONAL JTPA STUDY 



This chapter briefly discusses the setting in which the study was conducted and provides an 
overview of the research design. The National JTPA Study was conducted within an ongoing national 
program, which had already been in operation for approximately five years when the research started. The 
study was intended to measure the impact of local programs operating within the structure of existing rules 
(set at the federal, state, and local levels) on eligibility, permitted services, and desired post-program 
outcomes. Because the study had as a goal changing local program operations as little as possible, this 
institutional and programmatic setting was — in most respects — taken as a given. The research design 
developed to achieve the Department's goals was heavily influenced by this context. 

L The Setting for the National JTPA Study 

This chapter describes this setting in two ways. First, it outlines key feature: of the national JTPA 
system in place during the initial years of the study that affected both the research design and tiie selection 
of sites. Second, it briefly describes the sites chosen for the study. More detailed descriptions of the sites 
are contained in the Appendix and in Chapter 7. 

A. Key Features of the JTPA System That Influenced the Research Design 

Both the overall structure of the JTPA system and the specific requirements in Title II-A 
influenced the design of the study, local and state officials' reactions to it, and the implementation of study 
procedures. The following features of the JTPA system for Title II-A were in place during the period of 
site selection and study implementation and were especially important in shq)ing the research design.^ 

• JTPA is decentralized, fostering great diversity and a local orientation in 
SDAs. 

The Job Training Partnership Act, Public Law 97-300, was passed in October 1982, at a time 
when the federal government was attempting to lessen its involvement in social programs and to provide 

*Thc JTPA program has changed - through administrative action and statutory changes - since its initial 
enactment. The most important legislative changes were made during 1992 and went into effect on July K 1993. 
The description in this chapter focuses on the JTPA program as it operated during the late 1980s (when sites were 
involved with the study), but many of the key program features discussed in the chapter continue in place to the 
present. 
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additional authority to the states and localities. As a result, the Act establishes a decentralized decision- 
making and funding structure. States and service delivery areas (SDAs) make major decisions about 
program design and operations within the constraints of the statute. In this setting, the U.S. Department 
of Ubor did not feel it appropriate or permissible to mandate SDA participation in the study, and 
researchers had to contact and convince individual SDAs to participate. 

• JTPA is structured to involve multiple decision-makers at the local level. 

The Act requires the establishment of a Private Industry Council (PIC) for each SDA and directs 
the PIC and the chief elected official(s) in the area to share responsibility for decisions on the best use 
of funds within the local SDA. The Act also mandates that the PIC include representatives of economic 
development agencies, the public Employment Service, individuals with handicaps, labor, education, 
community-based organizations, and businesses - with representatives of the private sector in the 
majority. This official decision structure is often augmented at the local level by the involvement of 
service providers, SDA staff, and social service agencies. 

Each of these groups, regardless of whether it had an actual vote on the PIC, could influence local 
decisions such as whether to participate in the National JTPA Study, and each weighed the implementation 
process for the study and the long-term benefits against its own and its constituents' interests. 

• JTPA does not follow a single program model. Within the structure of an 
extensive list of permitted services and the federal eligibility guidelines, 
SDAs develop service plans to reflect local conditions and their perception 
of the needs of their clients (which could include both job-seekers and 
employers). 

PICs and local elected officials approve the job traiiiing plan for the SDA, making choices about 
the availability and emphasis of services and about recruitment priorities. Most types of employment and 
training activities arc permitted. However, public service employment ended with the enactment of JTPA, 
and subsidized work experience is limited to specific situations. At the time the study began, some SDAs 
included options for the provision of basic education skills and long-term training, while others 
emphasized shorter-term programs for the more job-ready, such as job search assistance and on-the-job 
training (OJT). The procedures for matching applicants with services have gradually evolved over time, 
but when the study began some SDAs based service recommendations on brief interviews, while others 
conducted muhi-day assessments with detailed testing of interests and skills. Based on this and the 
available service options in the SDA, staff and clients developed employability development plans, which 
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again varied in detail across sites. This plan was generally individualized in terms of the duration of 
services, number of steps, and sequence, and could be changed in response to changes in the person's 
needs, the failure of initial services to produce employment, or new service opportunities.^ 

This diversity of services and administrative procedures greatly complicated the development of 
service categories for which to estimate impacts and study procedures to introduce random assignment into 
the ongoing administrative process. 

• At the time the study began, many SDAs had to exert considerable efibrt 
to meet enrollment and expenditure goals, particularly for some tai^et 
groups. 

SDAs and states need to meet overall enrollment goals, goals for the enrollment of specific target 
groups, and expenditure plans.^ At the time, JTPA also required that not less than 40 percent of the Title 
II-A 78 percent funds be spent on eligible youth.^ 

During the period of site selection, recruiting was often a problem for SDAs. Because youths 
normally constituted less than 40 percent of the eligible population, local staff often faced a special 
challenge recruiting and enrolling enough youths to meet the 40 percent expenditure target. Furthermore, 
with the decrease in the unemployment rate in the mid and late 1980s, many SDAs and their service 
providers faced recruitment problems more generally as the more job-ready could obtain employment. 
Thus, the effort involved in recruitment and the characteristics of the applicant pool available to SDAs 
routinely changed for many different reasons. There was no unchanging picture of local JTPA applicants. 

Since JTPA funds are, on average, sufficient to provide services to less than 10 percent of the 
eligible population, the Department and the researchers expected that either (1) SDAs would have 
sufficient applicants to allow creation of a control group with no decrease in service levels, or (2) an 
increase in the level and quality of recruitment activities would produce the additional numbers needed 



^In some SDAs, assessment is viewed as a sequence of actions, with each phase producing decisions only about 
the immediate next step. Job search assistance was often used as a startinf; point and served as both an extended 
assessment and a labor market screen to identify the job-ready. 

'Section 141 of the Act specifies that the service delivery system is to provide employment and training 
opportunities to those who can benefit from, and who are most in need of, such opportunities. The system is also 
expected to make efforts to provide "equitable services" among "substantial segments" of the eligible population. 
Except for explicit requirements to serve dropouts and recipients of Aid to Families with Dependent Children 
(AFDC), these two terms are not defined in the Act or regulations. 

*Title II-A 78 percent funds refers to the 78 percent of Title II-A money distributed by fonrnula to states and 
SDAs. Beyond the spending requirement mentioned in the text, some states have rules to rec2q>ture funds if 
expenditures fall below a predetermined level, and others tie the allocation of incentive funds to expenditure ievels. 
Both factors increase the importance of SDAs' meeting planned enrollment and expenditure levels. 
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to maintain enrollment levels. However, the difficulty of JTPA recruitment led many SDAs to doubt they 
could attain the numbers of applicants needed to conduct random assignment.* 

. Performance standards and distribution of 6 percent incentive funds 
resul .ed in an emphasis on successful achievement of short-term outcome 
measures. 

JTPA has established an extensive system of performance standards, which awards incentive funds 
to SDAs that exceed their standards and provides for technical assistance or (in extreme cases) 
reorganization for those that do not. SDAs' performance for their enrollees is evaluated at the federal, 
state, and local levels, according to predetermined, outcome-based standards, and incentive funds are 
distributed based on the extent to which SDAs exceed their standards.*^ During the late 1980s, this led 
to an emphasis on successful achievement of short-term outcomes, a reliance on performance-based 
contracts, and, in many cases, some initial screening and services prior to the point of enrollment. 

In recruiting sites, the researchers found that states and SDAs varied in the importance they 
attached to meeting and exceeding specific standards of performance, but SDAs for which the awards were 
particularly important were less inclined to undertake new projects (including participation in the study) 
that might jeopardize performance outcomes.' Some states and SDAs encouraged the targeting of hard- 



»In a system with many more applicants than service "slots." the lottery-like random assignment process is usually 
seen as a fair way to allocate a scarce resource. However, when this initial surplus of applicants is not present, 
random assignment raises more ethical issues. j 

*From the inception of JTPA through program year 1987 (ending in June 1988), SDA performance for adults 
completing a JTPA enrollment ("terminces" in the parlance of JTPA) was judged by the following sumdards: the 
percentage of adults who obtained employment at tennination from the program, the percentage of adults who were 
receiving welfare when they enrolled in JTPA and who found a job at termination, the average wage at Pl«ernent 
in a job and the program cost per adult who obtained employment. In program year 1988, the Department added 
four port-program measures: the percentage of adults who were employed 13 weeks after termination, the percentage 
of adults receiving welfare at enrollment who were employed at 13 weeks after termination, the average number of 
weeks worked by all adults during the 13 weeks followi^ termination, and the average weekly eammgs of adults 
employed in the 13th week. For youth terminces, the original standards included the percent^e who entered 
employment, the percentage who entered employment or attained other quantifiable measures of program success 
(labeled a "positive termination"), and the program cost per youth with a positive termination. In program year 1 988, 
the Department added a fourth youth perf^ormance measure: employability enhancement, which counts only positive 
terminations other than employment. Starting in program year 1988, states were directed to choose 8 of the 12 
possible standards. (Since the period of random assignment for the study, there have been further changes m JTPA 
performance measures.) For each measure, the Department set national levels that - at state option - can be 
adjusted to reflect the characteristics of those served and the conditions in the local labor market. 

'Researeh done for the National Commission for Employment Policy found similar variations. See National 
Commission for Employment Policy, 1988. 
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to-serve subgroups. These special recruitment targets also complicated implementation of the study at the 
local level/ 

The performance standards, coupled with JTPA's rules limiting administrative expenditures to no 
more than 15 percent of spending, led to frequent use of performance-based contracts for service 
providers. Such contracts tie payment of funds to the achievement of pre-specified performance 
benchmarks, such as the achievement of competencies o\ ♦he placement of individuals in unsubsidized 
employment. The existence of these contracts added to SDAs' - and particularly to service providers' 
- concerns about the diversion of randomly selected individuals away from JTPA services. 

In sum, SDAs and the service providers they contract with viewed as a substantial risk the 
establishment of the control group and the consequent diversion of some of those applicants judged by 
them to be likely to succeed. 

B. An Overview of the Study Sites 

As shown in Exhibits 2.1 and 2.2, the 16 study sites' are spread throughout the nation, with 2 
in the Northeast, 4 in the South, 7 in the Midwest, and 3 in the West. Oakland and Jersey City are 
located in large metropolitan areas with substantial black, Hispanic, and other minority residents, but no 
large central cities are included among the sites. Large central cities often have many agencies handling 
recruiting and program intake, which would have made implementation of random assignment procedures 
very difficuk.^^ Among the sample are predominantly rural or small-town sites and mixed urban- 
suburban-rural sites. The largest cities within each of the 16 study sites range in population from Oakland 
(372,000 in 1990) and Omaha (336,000) to Thief River Falls, Minnesota (under 10,000). 

Profiles of each site, presented in the Appendirt, depict the SDAs and their local environments as 
of the time the sites were selected for the study and the random assignment process was going on 
(primarily program years 1986-88), and therefore do not reflect subsequent changes. Chapter 7 continues 
the discussion of the study sites by examining the process by which they were selected and discussing the 
extent to which they include the diversity of the JTPA system nationally. 



•For example, if an SDA set an cnrolhncnt target for a group at a much larger percentage than its incidence in 
the population, it would be under pressure to enroll every member of tliat group who could be recruited. 
'Also referred to in this report as the "study SDAs" or simply the "SDAs." 

^^or example, the City of Los Angeles program involved more than 50 service providers at the time of site 
selection for the study. Despite the site staffs initial interest in participating, it was not possible to solve the many 
logistical problems of doing the study within the available research budget. 



EXHIBIT 2.1 

LOCATION OF THE NATIONAL JTPA STUDY SITES 



OfCQON 



MONTAIM 



WYOUNO 



NORTH tMKOTA 



SOUTH DAKOTA 



112 



.UmNESOTAJ 



I VMOONSM 



UTAH 



10 

OOLORAOO 



NEBRASKA 



lUMOIS 



KANSAS 



MISSOURI 
■16 



KENTUCKY 



NO.CAAOUM 



MS 



TEXAS 



•O. 



OEOMIA 



1. Butte, Mont. 

2. C«dar Rapids, Iowa 

3. Coosa Vatley, Ga. 

4. Corpus Christi, Tex. 

5. Decatur, lU. 

6. Fort Wayne, !nd. 

7. Heartland, Fla. 

8. Jackson, Miss. 



SITE NAME 

9. Jersey City, N.J. 

10. Larimer County, Colo. 

11. Marion, Ohio 

12. Northwest Minnesota 

13. Oakland, Calif. 

14. Omaha, Neb. 

15. Providence. R.I. 

16. SpringtieU, Mo. 
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EXHIBIT 2.2 

KEY FACTS ABOUT THE NATIONAL JTPA STUDY SITES 



18-Month 

Census Study 
Site Name SPA Name Region Largest CiQ^ Sample Size 



Butte, Mont. 

Cedar Rapids, Iowa 
Coosa Valley, Ga. 
Corpus Christi, Tex. 

Decatur, 111. 

Fort Wayne, Ind. 
Heartland, Fla. 
Jackson, Miss. 
Jersey City, NJ. 

Larimer County, Colo. 
Marion, Ohio 

Northwest Minnesota 



Oakland, Calif. 
Omaha, Neb. 
Providence, R,L 
Springfield, Mo. 



Concentrated Employment 
Program, Mont. 
East Central Iowa 
Coosa Valley, Ga. 
Corpus Christi/Nueces 
County, Tex. 
Macon/De Witt 
Counties, III. 
Northeast Indiana 
Heartland, Fla. 
Capital Area, Miss. 
Corporation for Employ- 
ment and Training, Inc. 
Larimer County, Colo. 
Crawford/Hancock/Marion/ 
Wyandot Coimties, Ohio 
Northwest Minnesota 
(Crookston and Thief 
River FaUs) 
Oakland, Calif. 
Greater Omaha, Neb. 
Providence/Cranston, R.L 
Job Council of the 
Ozarks, Mo. 



West 

Midwest 

South 

South 

Midwest 

Midwest 
South 
South 
Northeast 

West 
Midwest 

Midwest 



West 
Midwest 
Northeast 
Midwest 



Butte 

Cedar Rapids 
Rome 

Corpus Christi 

Decatur 

Fort Wayne 
Lakeland 
Jackson 
Jersey City 

Fort Collins 
Marion 

Thief River 
Falls 

Oakland 
Omaha 
Providence 
Springfield 



477 

346 
1,806 
1,498 

471 

2,559 
597 
1,375 
1,170 

668 
1,083 

498 



1,048 
956 
1,277 
1,202 



All Sites 



17,031 



n. An Overview of the Study Design 

The Department's choice of random assignment as the method for estimating program impacts was 
merely the first of many complex research design issues faced on the project. The particular random 
assignment research design and study sites chosen determined the impact questions that the research could 
address. The Department's multiple goals for the study presented a challenge in developing the research 
design. These goals, as mentioned earlier, were to (1) select sites that represent the diversity of the JTPA 
system, (2) estimate overall impacts for JTPA programs in the sites for the full sample and for key 
subgroups, (3) estimate impacts for specific service strategies anchored in classroom training in 
occupational skills (CT-OS) and on-the-job training (OJT)" for the full sample and for key subgroups, 
and (4) assess the impact of local programs that were following their normal practices as much as possible. 

A. Striking the Balance Among the Re search Goals 

Ideally, the first goal would have been achieved by randomly selecting local programs to include 
in the study. However, the Department did not choose to emphasize this goal to the extent of mandating 
participation in the study. The decision to rely on volunteer sites was also reinforced by the need to have 
sites interested in and able to implement the complex research design. As Chapter 7 discusses in more 
detail, it did not prove feasible to get a random sample of local JTPA programs to agree voluntarily to 
participate in the study. This occurred in part because of the other study goals; a complex random 
assignment design and a large sample (implying a relatively long period of random assignment) were 
needed to answer the various impact questions for the full sample and key subgroups. As Chapter 7 
discusses, the sites selected for the study do include much of the diversity of the JTPA system and are 
in many ways similar to the national average for all SDAs. 

The tensions between the remaining three research goals led to a fairly complex random 
assignment research design. The research plan, shown in Exhibit 2.3, introduced random assignment into 
the application process with as little disruption as possible while still producing (1) an estimate of the net 
impact of overall JTPA services in the study sites, and (2) estimates of the net impact of each service 
strategy for the people local staff found appropriate for it. Random assignment took place after local staff 
made their service recommendations and designated a service strategy for the enrollee. 



"In JTPA, each enrollee is active in a particular activity such as OJT or CT-OS. For purposes of the research, 
as discussed in Chapter 5, these recommendations were grouped into "service strategics," which were used for 
estimating subgroups for program impacts. 
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This design addressed the two central impact questions: 

• Overall program net impacts were estimated by combining the three 
treatment groups — recommended for the classroom training, OJT/JS A (on-the- 
job training/job search assistance), and "other services" strategies — and 
comparing their experience following random assignment with a combined 
control group made up of the three separate control groups. 

Service strategy net impacts were estimated by comparing the experiences of 
the treatment group for each strategy with those of the control group for that 
strategy. 

The design used in this study preserved to a great extent local staffs' ability to operate their programs 
following normal practices as to intake, assessment, and service referral. However, it did not produce 
experimental comparisons of the relative (or "differential") impacts of service strategies because 
assignment to each of the three treatment groups and the control group was not random. It was expected, 
and was the case over the implementation of the study, that the groups recommended by SDA staff for 
the three service strategies differed on many baseline characteristics.^^ Therefore, because those referred 
to the three service strategies differed initially on many characteristics in addition to the services to which 
they were given access, direct experimental comparisons of the impact of service strategies were not 
possible.'^ This choice of research design rested on an explicit decision by the Department and the 
researchers that testing JTPA as closely as possible to the way it normally operates was more important 
than producing differential impacts for the service strategies. 

Once this choice was made, the researchers faced the issue of when in the client flow to do 
random assignment, which also involved balancing research goals. The goal of estimating the overall 
effect of all program services leads to a desire to place random assignment as early in the intake process 
as possible; this will lessen the control group's contact with the program. Yet local staff had to work with 
a client long enough to make recommendations for services and designate a service strategy before random 
assignment could take place. 



*^See Keniple, Doolittle, and Wallace, 1993, and Bloom ct al, 1993, for a discussion of these differences. 

^^Obtaining experimental estimates of the differential effect of one service strategy versus another would have 
required randomly assigning individuals to a service strategy, and thus overriding any local assessment procedures. 
This would have been clearly inconsistent with the goal of changing local program operations as little as possible. 
Many people would have been assigned to different services than local staff would have recommended following their 
normal assessment. Since individualized service plans based on client interests and needs are one objective of JTPA, 
this research design was not acceptable. 



There was another relevant factor: As mentioned in more detail later in this report, at each stage 
in program intake, applicants can "exit" the system by deciding not to continue the process for a variety 
of reasons.^^ Therefore, the closer random assignment is placed to the actual start of services (i.e., the 
later in the intake process), the more the research sample resembles those who are actually enrolled in 
JTPA services and the higher the proportion of the treatment group receiving services. This makes the 
comparison of the experiences of the treatment group to the control group closer to a test of the impact 
of service receipt as opposed to the offer of services. A final factor was the burden of local program staff; 
the earlier random assignment was placed in the intake process, the fewer steps local program staff had 
to complete for the entire sample before the control group was identified. 

The final research design struck a balance between these competing factors. The researchers 
worked with staff in each site to identify an early point in JTPA client flow at which local staff believed 
they could decide whether an applicant was appropriate for JTPA and identify the appropriate services and 
service strategy. In most cases, this came at the end of the normal assessment, which typically occurred 
fairly early in the application process. 

B. Data Sources, Key Outcomes, and the Impact Sample 

The analysis of JTPA program implementation and impacts rests on seven data sources: 

• Background Information Form (BIF). Applicants for JTPA completed the 
BIF with the assistance of site staff prior to random assignment. It provides 
basic demographic, education, employment, household, and public assistance 
information on 99.5 percent (20,501 individuals) of the experimental sample. 
In addition to this, site staff also recorded on the BIF their JTPA service 
recommendations for applicants, the corresponding service strategy for the 
research, and the outcome of random assignment (treatment or control status). 

• Random Assignment Record. This computer file contains information 
collected when site staff placed the phone call to MDRC during which 
individuals were randomly assigned to the treatment or control group. It 
includes key identifiers from the BIF, the staff recommendation for individual 
services and the corresponding research service strategy, and the outcome of 
random assignment. Sinre random assignment could occur only through such 
a phone call, and all information had to be provided prior to assignment, this 
file includes all 20,601 members of the experimental sample and has no 
missing data. In the few cases where the random assignment record and BIF 



^^Among other possible reasons, clients might find a job, decide they were not interested in the available JTPA 
services, lose interest in training, or be dissuaded from continuing by the actions of JTPA staff. 
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differed on research status (because site staff incorrectly recorded treatment or 
control status during the phone call), the information in the random assignment 
record was used. 

JTPA Enrollment, Participation, and Termination Data. SDA staff 
provided this information from their existing automated management 
information systems for all members of the sample.*^ Sites provided 
information on any JTPA enrollments for members of the experimental sample 
during the 18 months following their date of random assignment,*^ Data 
elements used in the study included enrollment start and end dates and the title 
of JTPA under which the person was enrolled (Title II-A and, in some cases, 
other titles as well), the activity into which the person was enrolled and its start 
and end dates, and program termination date and status (e.g., employed or not). 
This information was used to identify initial and subsequent enrollments in 
JTPA, the number and types of activities in which enrollees participated, and 
the length of enrollments. 

• Follow-up Survey. For the interim impact report, post-random assignment 
outcome data was collected through an interview with a subsample of those 
randomly assigned (see discussion below for the details) conducted between 13 
and 22 months following the date of their random assignment. This variable 
length of follow-up was used to shorten the time during which the survey was 
fielded in order to lower the costs of data collection.*^ It provides 8 
continuous self-reported history of employment and spells of education and 
training lasting a week or longer. E)etailed characteristics of each job (hours 
worked, wage rate, overtime, and bonus pay) were collected as well as the type 
of education or training and the number of hours participating. A second 
Survey for a subsample of those randomly assigned will also be used in the 
final impact report. 

• Earnings Data from State Unemployment (UI) Insurance Agencies. State 
Unemployment Insurance agencies collect quarteriy data on wages and salaries 



*^In some cases, these were systems operated by states. Since each SDA had a different automated system, the 
information needed for the study was extracted and modified into a standard record format. 

*^About one-third of the members of the treatment group never were enrolled in JTPA in the study sites during 
the 18-month follow-up period, so there arc no enrollment or participation data for these persons. In addition, as 
planned, virtually all members of the control group were never enrolled in JTPA so there are no data for them. 

"The original plan was to attempt to interview all sample members approximately 18 months after the date of 
their random assignment. Since nearly two years elapsed between the start of random assignment in the first site 
and its end in the last site, the survey staff would have been fielding the survey for a similar period. Under the 
revised plan, early cohorts in the sample were interviewed more than 18 months after their date of random 
assignment, while late cohorts were interviewed fewer than 18 months after this event. This shortened the period 
during whidi the interview was fielded but also (since a complete post-random assignment history was collected) 
resulted in variable lengths of follow-up. This is discussed further in Chapter 3. 
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for most workers.** Data from states for 14 of the 16 sites were used For two 
purposes: (1) pre-random assignment wage data were used as a baseline 
variable in the estimates of impacts, and (2) post-random assignment data 
(quarters 1 to 4) were used to test for - and, for the aduh female group, 
compensate for — nonresponse bias in the survey data on employment and 
earnings.*' In the final report, these data will also be used as a central 
outcome in the impact analysis, because the second survey will only be 
conducted for a subsample. 

• Administrative Records on JTPA Programs. As part of the administration 
of JTPA, local programs collect and report to states and the federal government 
information on the number and characteristics of the people they serve, on their 
success in meeting performance standards, and on program costs.^^ These 
were used to obtain background infonnation on the 16 study SDAs and to 
compare them to all SDAs in the national JTPA system. 

• Field Research on the Study SDAs. During site selection and implementation 
of the study, researchers collected a range of information from sites including 
program plans, recruiting materials, lists of service providers, and samples of 
typical contracts. The team also documented the SDAs' organizational 
structure and client flow in order to design q)propriate random assignment 
procedures.^* 

In calculating the impact estimates for the study, the Department and the researchers chose to 
focus on outcomes closely related to the statutory goals of the program, many of which are included in 
the existing performance standards system.^ The 18-month impact report^ focuses on the program's 
success in increasing employment rates and earnings, two explicit statutory goals. The final impact report 
will add analysis of the program's success in reducing the receipt of public assistaiice, the third explicit 
statutory goal.^* Reflecting the congressional intent that JTPA services should be seen as an investment 
in the skills of participants, the study will summarize its findings within a benefit-cost frameworic, which 



^•Exceptions are individuals working for the federal government, railroads, and agriculture; the self-employed; 
and those woricing "off the books" or in fims not paying taxes. 

^^sable UI wage data had not been obtained from Ohio or New Jersey at the time the analysis was done, and 
data from Montana were incomplete. 

^^These are contained in the JTPA Annual Status Report (JASR) data and in reports by individual states and 
SDAs. 

Staff did not have the resources to analyze the nature and quality of the services provided by the many 
organizations under contract to the SDA. Information on services received by sample members primarily came from 
each site's data on JTPA enrollment and participation. 

^^The crucial difference is that the performance standards system measures termination (or, more recently, post- 
program) outcomes rather than program impacts, which are estimated in this study. 

"Bloom ct al., 1993. 

^^The final report will rely on administrative records on welfare receipt in a subset of the study sites, 
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assesses the program from the perspectives of government budgets, program participants, and society as 
a whole. 

The overall experimental sample is 20,601, while the 18-month impact sample included 14,441 
individuals. The difference between the total number randomly assigned and the smaller impact sample 
for the 18-month report arises because of the exclusion of three groups: 

• 473 "extra" treatment group members who had been randomly assigned at a 
ratio higher than 2 to 1 (to ease the recruitment burden on sites) were excluded 
from the analysis so that impact estimates would not involve complex weighing 
procedures. 

3,087 people scheduled for follow-up interviews prior to 18 months after 
random assignment were excluded to avoid the complexities introduced by 
variable-length follow-up. 

2,584 people (15 percent of the attempted surveys) were excluded because they 
did not respond to the 18-month survey. 

C. Interpreting the Impact Findings 

While choosing a random assignment impact design helps in providing internally valid estimates 
of the impacts of the JTPA program as it operated in the sites during the period of the study, it does not 
allows researchers to address every conceivable question simultaneously. In interpreting the impact 
findings presented in the companion volume (the 18-month impact report), the reader must be aware of 
two key aspects of the questions addressed: the nature of the service comparison for which impacts are 
estimated and the possible link between labor market conditions and program impacts. 

1. The Service Comparison . The comparison of the treatment and control group outcomes 
used to estimate program impacts is a test of the additional employment-related services the treatment 
group received above those received by the control group, rather than a test of JTPA services versus no 
services of any kind. Under the final design for the study, these control group services could take two 
forms: (1) All members of the control group did receive a JTPA assessment prior to random assignment 
so that local staff could recommend a service strategy, and (2) members of the control group could seek 
services from non-JTPA sources.^' 



^^Chapter 5 discusses the extent to which they received such services and the resulting difference in services 
between the treatment and conti'ol group. 
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Furthermore, it is incorrect to interpret this comparison as the difference the existence of JTPA 
makes. If the JTPA program did not exist, other programs might well change their service offering, and 
some nonrandom subsample of those served by JTPA would most likely seek and receive services from 
other sources. However, there is limited funding for services in the non-JTPA programs, so some people 
who would otherwise be served by these other programs if JTPA did not exist would have to be turned 
away. Because of this, the control group experiences in a world with JTPA does not represent what would 
have happened to them had the JTPA program not existed. More control group members are likely to be 
served in other programs in an experimental impact study of JTPA than would be the case if JTPA did 
not exist. This service substitution leads to an underestimate of the impact of "the existence" of JTPA and 
its services. Instead, the impact estimates in this study should be interpreted as the difference in the 
outcomes produced by the observed difference in employment and training services between the treatment 
and control group. 

One particular type of service the control group is likely to receive deserves special mention in 
this context. Although data are not available, it is probably true that many people who applied for JTPA 
in all the study SDAs had already contacted their local Job Service office about employment 
opportunities.^^ Thus, access to the job listing assembled by the Job Service should be assumed to be 
part of the "background level" of services against which JTPA services for the treatment group are 
assessed. In three sites (Butte, Fort Wayne, and Northwest Minnesota), JTPA and the Job Service were 
co-located during much of the period of random assignment, so this presumption is clearly true in these 
sites.^^ 

2. The Link Between Labor Market Condittons and Impacts ^ Because the treatment and 
control group in each site were seeking jobs and woricing in the same labor market at the same time, many 
analytical problems arising because of differences in labor market demand and supply over time and across 
sites were avoided.^* However, some such issues remained. 



^^In one SDA in the study that collected this information, about 75 percent of applicants had already registered 
with the local Job Service. The SDA required applicants who had not registered to do so. 
"This arrangement is discussed in the relevant site profiles in the Appendix. 

^'For example, in a research design comparing the experiences of "similar" individuals before and after a new 
program is introduced, researchers must try to statistically adjust for differences in the labor market in the two 
periods. In a comparison group design where individuals served in a program arc compared with similar individuals 
in another site not served by a program, researchers have attempted to select the two comparison sites so they are 
initially similar. However, over time labor maricct differences often emerge, and researchers have attempted to make 
statistical adjustments for this. Both problems arc avoided by random assignment. See Wiseman and Greenberg, 
1992, for a discussion of several issues related to these types of design. 
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a. Site DifTerences in Labor Market Conditions. First, it is common in multi-site studies to 
find differences in impacts among the sites, and this was the case in the JTPA study. Since many factors 
potentially related to impacts (such as JTPA program characteristics, labor market conditions, service 
availability outside of JTPA, and client characteristics) varied among the relatively small number of sites 
in the study, it was difficult to isolate each factor's effect on program impacts. Despite these difficulties, 
because of the interest in understanding the link between site characteristics and impacts, the researchers 
undertook an analysis of this issue.^^ 

While the results of the analysis are reported in the 1 8-month impact report, it is useful to describe 
the research approach taken. Before examining site-level impacts, the team identified factors (including 
local labor market conditions) for which there was a theoretical reason to expect an effect on impacts.^° 
It was important to begin with a plausible hypothesis because a continued search for possible explanations 
could eventually find a statistical model able to explain much of the variation in site impacts, even if no 
causal link existed. The team then estimated a regression model with site impacts as a function of the 
baseline characteristics of JTPA participants in the sample, the local environment (including labor market 
conditions), and key features of the local JTPA program. Though there are many limitations to this 
analysis, which are discussed in the 18-month impact report, it does provide a chance to identify factors 
that have a clear effect on local program impacts. 

b. Possible Displacement of Other Employees. One of the central goals of the JTPA impact 
study was to estimate the extent to which the program leads to increased employment and earnings. This 
analysis rests on a comparison of the employment history of the treatment and control groups. However, 
it is possible that members of the treatment group could find jobs that would have otherwise been filled 
with job-seekers from outside the treatment group and that the latter individuals would, as a consequence, 
become unemployed or accept lower-wage jobs (i.e., be "displaced"). If this occurred, a comparison of 
the employment and earnings of the treatment and control groups would overestimate the employment 
benefits of the program from a societal perspective. 

Evaluators have not found any satisfactory way to address this displacement issue, which arises 
whether program impacts are calculated using random assignment or nonexperimental techniques. The 
situation is even more complex because there is also the theoretical possibility of "replacement" of 



^'This analysis is reported in Chapter 7 of the 18-month impact report: Bloom et al., 1993. 
^^e selection of factors and the way in which the factors were measured drew heavily on the implementation 
research reported in Kemple. Doolittle. and Wallace. 1993. 
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workers; this would occur if program participants moved from industries or occupations in which labor 
surpluses exist to those with shortages, thereby eliminating bottlenecks and allowing production to 
increase. In this situation, the usual impact estimates would underestimate benefits from a societal 
perspective.^^ Some economists have argued that since proper macroeconomic policy could lead to 
noninflationary full employment, the benefit-cost analysis of individual programs need not examine 
possible displacement effects.^^ 

Because of the complexity of this issue, most evaluations of employment-related programs do not 
attempt to estimate the extent of displacement of workers from jobs." The National JTPA Study follows 
this convention in its calculation of program impacts and benefits. Similarly, the study did not attempt 
to estimate the extent of replacement of workers. 



^^^et Betsey et al, 1985, p. 150; Johnson, 1979; and Kemper, 1980, for examples of this argument. 
'^Gramlich, 1981, is often cited for this proposition. 

"See Betsey et al., 1985, for a discussion of several studies that did try to estimate displacement. Most have 
concerned employment displacement in the public sector brought about by public service employment programs. 
Betsey et al. concluded that these efforts to estimate displacement do not lead to convincing conclusions. 
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CHAPTERS 
THE JTPA PROGRAM IN THE STUDY SITES 



JTPA, by design, is not a single program implemented uniformly across the country. While the 
federal statute and regulations do provide a basic structure for the program (as to eligibility, permitted 
services, etc.), states and especially SDAs make many of the policy and management decisions. Not 
surprisingly, this results in great variation in many key aspects of the program. Furthermore, since SDAs 
often rely primarily on other organizations for many of the client services JTPA finances, the nature of 
the services provided also reflects such things as the size and stability of local education and training 
organizations. 

This chapter provides background on many of the key program design choices made by the study 
sites on administrative structure, service providers, and service emphasis. Since there is no comparable 
information available for SDAs nationally, it is impossible to assess how the study sites compare to the 
national program. But the material presented herein illustrates that the study sites, because of the choices 
they have made, include examples of many different types of administrative and programmatic 
approaches.' This chapter is also useful background for understanding and interpreting the impact 
findings from the study because it begins the description of the program services for which impacts are 
estimated. 



I. The Administrative Structure of the SDAs 

The federal statute assigned responsibility for the delivery of services funded under Title II-A of 
JTPA to local SDAs, under a "partnership" between government and business. But much was left to 
negotiations at the local level. Specifically, negotiations would determine which entity - the government, 
the Private Industry Council (PIC), or some combination of both - would play the following roles: 

Funding recipient, which received and was held legally responsible for the 
JTPA funds allocated to the SDA by the state. 

Administrative body, which administered the program, including developing 
tlie SDA's service plan, selecting contractors, recruiting clients, and arranging 
services. 



'This chapter draws on material in Kemple, Doolittle, and Wallace, 1993. 
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If local negotiations determined that the PIC would play either or both of these roles, then the PIC would 
have to formally incorporate (in order to receive public funds) and hire staff to conduct the business of 
the PIC. Even in situations in which PICs did not perform either of these fijnctions, they often established 
their own identity separate from the government by incorporating and/or hiring their own staff. 

The choices that the government and PICs made in the 16 SDAs in the JTPA study are shown 
in Exhibit 3.1. In six SDAs (labeled "government-operated"), the local government (or consortium of 
contiguous governments) acted as both funding recipient and administrative body, and the PIC chose not 
to incorporate or to have a separate PIC staff. In another three SDAs (labeled "PIC-operated"), the PIC 
was an incorporated body with its own staff and served as administrative body and funding recipient. In 
the remaining seven SDAs, the government and PIC shared responsibilities and/or had various staffing 
arrangements. 

n. The Setection and Role of Service Providers 

The federal statute also assigns to the SDA the responsibility for selecting and defining the roles 
of JTPA service providers, which can range fircm public agencies and community-based and other 
nonprofit organizations to private for-profit companies.^ As Exhibit 3.2 shows, during the period of 
random assignment, service providers for employment, training, and education services varied across the 
16 study SDAs. Perhaps surprisingly, the arrangement of OJTs - i.e., the placement of clients in OJT 
slots in subsidized training in either the private or public sector — involved private for-profit agencies in 
only 2 SDAs. In 1 1 SDAs, OJT was run directly by SDA staff, with the state Employment or Job Service 
- a traditional provider of OJT - also playing a role in 3 of these SDAs. Exhibit 3.2 also shows that 
public education institutions — vocational-technical schools, community colleges, or universities — were 
providers of classroom training in 14 sites; and that proprietary schools were providers in 8 of the 16 
SDAs. Further, job search assistance was provided by a range of agencies, but most typically by either 
the SDA or the Job Service, or both. Basic education, when it was offered, was provided by public 
education in 9 of 12 cases. But it was not offered in 4 SDAs, at least not as an identifiable stand-alone 
activity. 

The provision of recruitment and assessment was largely an SDA staff function, rather than being 
contracted out to other organizations. SDAs were responsible for most or all recruitment activities in 10 



^Through its state coordination and special services plan, the state can influence this decision. 
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ORGANIZATIONAL STRUCTURE OF THE SDAs PARTICIPATING 
IN THE NATIONAL JTPA STUDY, BY SITE 

SDA Organizational Structure Funding Administrative Incorporated Separate PIC 
and Site Recipient Body PIC? Staff? 



Government-Operated 



Butte, Mont. 


Gov. Entity (a) 


Gov. Entity (a) 


No (a) 


No (a) 




Gov. Entity 


Gov. Entity 


No 


No 


Decatur, 111. 


Gov. Entity 


Gov. Entity 


No 


No 


Heartland, Fla. 


Gov. Entity (b) Gov. Entity (b) 


No (b) 


No (b) 


Omaha, Neb. 


Gov. Entity 


Gov. Entity 


No 


No 


Springfield, Mo. 


Gov. Entity 


Gov. Entity 


No 


No 


PlC-Ooerated 










Fnrt Wavne Ind 


PIC 


PIC 


Yes 


Yes 


^4i>rinn Ohio 


PIC 


PIC 


Yes 


Yes 


Northwest Minnesota 


PIC 


PIC 


Yes 


Yes 


Govemment/PIC-Operated 










Coosa Valley, Ga. 


n/a (c) 


n/a (c) 


Yes 


No 


Corpus Christi, Tex. 


PIC 


Gov. Entity/ 


Yes 


Yes 






PIC (d) 






Jackson, Miss. 


Gov. Entity 


Gov. Entity 


Yes 


No 


Jersey City, N.J. 


Gov. Entity 


Gov. Entity (e) 


Yes 


Yes 


Larimer County, Colo, (f) 


Gov. Entity 


Gov. Entity 


No 


No 


Oakland, Calif. 


Gov. Entity 


Gov. Entity/ 


Yes 


Yes 






PIC (g) 






Providence, R.I. (f) 


Gov. Entity 


Gov. Entity 


No 


No 



SOURCE: Information collected by MDRC site representatives during the National JTPA Study. 

NOTES: (a) On July 1, 1990, the operator of this SDA changed from the Montana Department of Labor 
and Industry, a government entity, to Montana Job Training Partnership, Inc., a private nonprofit organization 
formed by the two PICs in Montana. At this time, the PIC became incorporated and developed a separate staff. 

(b) In December 1989, the PIC became the funding recipient and administrative body; in January 
1990, it became incorporated and developed a separate staff. 

(c) Both the funding recipient and the administrative body are the Coosa Valley Area Regional 
Development Center. This organization was established by state law with the consent of the PIC 

and the local government entity responsible for JTPA. 

(d) A government entity was originally responsible for administration; on July 1, 1988, 
the PIC began to function as both the funding recipient and the administrative body. 

(e) The Corporation for Employment and Training, Inc. (CET), was contracted by Jersey City 
to act as the administrative body. CET lost its contract with the city at the end of June 1990. 

(f) According to the criteria used in this table, this site should be included in the government- 
operated SDA category; however, MDRC site representatives included it in the govemment/PIC- 
operated SDA category because the PIC plays an especially significant role in deciding how to spend 
JTPA funds and choosing service providers. 

(g) The city of Oakland and the PIC share this role. 
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of the 16 SDAs, while service providers were responsible for most or all recruitment in 4, and in 2 SDAs 
the function was shared equally. Most or all of the client screening and assessment were done by SDA 
staff in 9 sites, and by service provider staff in 4. In the remaining 3, both organizations played a role. 

SDAs had a choice of structuring their legal relationship with service providers in one of two quite 
different ways: 

• Cost-reimb'irsement contracts, in which the contractor was paid for services 
rendered regardless of the outcome for the enroUees. 

• Fixed unit price, performance-based contracts, in which full payment was, 
by law, contingent on enrollees' achieving specific outcomes such as job 
placement in a training-related job at a certain wage rate for a specified period 
of time. 

SDAs* choice of performance-based contracts has provoked considerable controversy. They were 
authorized to provide SDAs with a mechanism to create incentives for service providers to meet the 
performance standards faced by the local program, but at the same time they can create incentives for the 
contractor to screen applicants carefully before accepting them, because of the risk of financial loss.^ 
There was, however, a major financial incentive for SDAs to adopt fixed unit price contracts: All costs 
of these contracts could be charged to the "training" cost category and none to administration, which had 
a ceiling of 15 percent of all local expenditures and was the source of funding for local planning, 
management, and other administrative tasks. 

As Exhibit 3.3 shows, the SDAs varied considerably in the percentage of training costs spent 
through performance-based contracts. Seven SDAs did not use performance-based contracts at all, and 
an eighth used them only slightly. At the other extreme, three SDAs (Coosa Valley, Jackson, and 
Northwest Minnesota) used performance-based contracts for more than 50 percent of both aduh and youth 
training expenditures. 

A further important choice facing SDAs was whether to (1) provide classroom training in 
occupational skills through referrals of individuals to training providers, (2) purchase a "class" for several 
people at once through a contract with a service provider, or (3) pursue both approaches. The 16 SDAs 
in the study showed considerable diversity in how they made this strategic choice during program year 
1988: 



'This risk was alleged to discourage service providers from accepting hard-to-serve clients. 
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EXHIBIT 3.3 



LEVEL OF USE OF PERFORMANCE-BASED CONTRACTS 
AMONG SITES IN THE NATIONAL JTPA STUDY, 
FOR ADULTS AND YOUTHS 





Percentage of Adult 




Percentage of Youth 




Training Expenditures 




Training Expenditures 


Level of Use 


Re^iiltinp from Performance*" 


Level of Use 


Resulting from Performance- 


and Site 


Based Contracts (a) 


and Site 


Based Contracts (a) 


Hieh Use (b) 




Hich Use (b) 




Coosa Valley, Ga. 


83 


Jackson, Miss. 


80 


Northwest Minnesota 


61 


Coosa Valley, Ga. 


80 


Jackson, Miss. 


60 


Northwest Minnesota 


68 




Oakland, Calif. 


65 


Moderate Use (c) 




Corpus Christi, Tex. 


51 


Corpus Christi, Tex. 


46 


Moderate Use (c) 




Omaha, Neb. 


38 






Oakland, Calif. 


37 


Omaha, Neb. 


28 


Providence, R.I. 


25 


Heartland, Fla. 


21 


Heartland, Fla. 


22 


Providence, R.I. 


18 


Little or No Use (d) 




Little or No Use (d) 




Butte, Mont. 


2 


Butte, Mont. 


0 


Cedar Rapids, Iowa 


0 


Cedar Rapids, Iowa 


A 

u 


Decatur, 111. 


0 


Decatur, 111. 


0 


Fort Wayne, Ind. 


0 


Fort Wayne, Ind. 


0 


Jersey City, N.J. 


0 


Jersey City, N.J. 


0 


Larimer County, Colo. 


0 


Larimer County, Colo. 


0 


Marion, Ohio 


0 


Marion, Ohio 


0 


Springfield, Mo. 


0 


Springfield, Mo. 


0 



SOURCE: MDRC calculations from program year 1988 fiscal records collected by Abt Associates Inc. 
from the 16 SDAs. 



NOTES: Under a JTPA performance-based contract, service providers can receive partial payments 
only when they atuin performance benchmarks. Also, full payment under these contracts is 
contingent upon three conditions: completion of training, placement in a training-related 
job, and receipt of a specified wage. All payments made under these contracts can be 
charged to training expenditures. 

(a) In some SDAs» the total for training expenditures includes some JTPA *8 percent" education 
funds, which are spent in the SDA but are not under the direct control of SDA staff. This may affect the 
percentages listed » but does not change the composition of the level of use categories. 

(b) A site had "high use* of performance-based contracts if payments under this type of 
contract accounted for over 50 percent of total Title II- A training expenditures. 

(c) A site had "moderate use* of performance-based contracts if payments under this type of 
contract accounted for 15 to 49 percent of total Title II -A training expenditures. 

(d) A site had • little or no use* of performance-based contracts if payments under this type 
of contract accounted for under 15 percent of toUl Title II-A training expenditures. 



Six SDAs relied exclusively on individual referrals, and tended to emphasize 
classroom training in occupational skills less than other sites. These included 
three rural sites (Butte, Marion, and Northwest Minnesota), which relied 
primarily on public vocational/technical institutes or other local public technical 
or community colleges, and three mixed urban/rural sites (Fort Wayne, Decatur, 
and Cedar Rapids), which relied primarily on similar types of institutions/ 

• Four SDAs relied exclusively on contracts to purchase a class. The four 
(Heartland, Jackson, Omaha, and Providence) were in urban areas and wrote 
from five to nine contacts within a program year. Occupations included truck 
driving, security guard, retail sales, automotive maintenance, food preparation, 
marketing, clerical skills, photocopy repair, and home health aide. Two SDAs 
with a much higher-than-average enrollment in classroom training in 
occupational skills (Jackson for adults, and Omaha for all target groups) fell 
into this category. 

• The remaining six SDAs used a mixture of individual referrals and class 
contracts. Among these SDAs, those in larger, urban areas (Corpus Christi, 
Jersey City, and Oakland) relied on community-based organizations for training 
contracts in addition to public vocational/technical institutes and colleges.^ 



m. Service Emphasis 

The service emphasis of SDAs can be described by examining the JTPA experience of the 18- 
month follow-up sample treatment group members who were enrolled in JTPA.^ Exhibits 3.4 and 3.5 
show the percentage of JTPA enrollees in the treatment group of the 18-month follow-up sample who 
were active in on-the-job training (OJT), classroom training in occupational skills (CT-OS), job search 
assistance (JSA) only, educational activities without training, and other non-training activities.^ 



'^Unfortunately, the enrollment and participation data collected from most sites did not include information on 
the occupations in which people were trained, and the multiplicity of individual referrals prevented the research team 
from collecting contracts for each training placement, which would have identified occupations. 

^As was true for the first group, data limitations prevent a complete discussion of the training occupations. 
However, the available information on training class contracts suggests that this type of training was for occupations 
similar to those in SDAs in the second group. 

^As discussed later in this report, about one-third of the treatment group never were enrolled in JTPA during the 
18-month follow-up period. The analysis of JTPA service receipt in this section focuses on the part of the treatment 
group that was enrolled. 

^Thcse categories are mutually exclusive, and a category for each sample member was chosen based in part on 
a hierarchy of services. Classroom training in occupational skills and on-the-job training were the most intensive, 
employment-related services typically offered by the sites, and they were placed at the "top" of this hierarchy. Since 
it was rare for a person to receive both of these expensive services, the choice of which was at the "top" of tlie 
service list does not affect the results in the exhibits materially; classroom training in occupational skills was placed 
first, because it is the most common intensive service. If a person was enrolled in JTPA classroom training in 
occupational skills (alone or in combination with anything else), then he or she was counted as being in the CT-OS 
category. Enrollment in on-the-job training was second in line in the hierarchy. Job search assistance as a sole 
service (JSA only) is clear-cut in meaning and was used as a category because it represented a conscious choice by 

(continued...) 
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The sites are ranked from highest to lowest percentage receiving OJT, and the differences across 
SDAs in sen'ice emphasis are striking* Further, some SDAs (most notably, Northwest Minnesota, Coosa 
Valley, Cedar Rapids, and Fort Wayne) consistently emphasized OJT for all four of the groups listed in 
the exhibits. SDAs also showed great differences in the proportion of their enroUees participating in job 
search assistance as a sole service; in some SDAs, this amounted to more than half of all enrollees for 
some target groups (e.g.. Fort Wayne for adult males and Heartland for all males), while in other SDAs, 
job search assistance was never provided as a sole service. Basic education not linked to training was very 
rare in most SDAs, but Marion emphasized it for all groups and Jackson for youths. 

Interestingly, two factors often cited as strong influences on service emphasis do not explain the 
diversity in service emphasis seen among the study SDAs. It is often assumed, for example, that the level 
of unemployment in an SDA is a major determinant of service emphasis. One hypothesis is that more 
classroom skills training would be offered in areas with a higher unemployment rate, because immediate 
employment opportunities, often seen as a prerequisite for on-the-job training and job search assistance, 
are more limited. This does not seem to have been the case in the study sites; the unemployment rate 
does not seem to be consistently related to a particular service emphasis among the 16 study sites.* 

The three sites with the lowest unemployment rates - Cedar Rapids, Omaha, and Providence - 
had very different mixes of OJT, CT-OS, and JSA only, and of these only Cedar Rapids placed a strong 
emphasis on on-the-job training. In addition, within each of these sites, there was considerable variety 
in service emphasis among the four target groups. Similarly, no consistent pattern can be seen in the 
service emphasis of the SDAs with the highest unemployment rates - Corpus Christi, Decatur, and 
Heartland - with the exception of low rates of OJT participation for female youths. 

Another hypothesis states that rural areas focus on OJT because they are unable to provide 
classroom training owing to the lack of service providers and/or a critical mass of clients needed to create 
and sustain a skills training class in a particular occupation. The inverse of this proposition holds that 



'(...continued) 

the SDA and/or its service providers to work for an unsubsidizcd job placement without providing more intensive 
services. The 1992 amendments generally prohibit this practice. Non-training education was used for individuals 
who were enrolled in activities such as adult basic education but never made the transition to training activities, 
which were more closely linked to job placement. TTiose who made the transition would be in OJT or CT-OS, as 
appropriate. Miscellaneous non-training services was the category used for people who were in activities such as 
pre-employment skills training or tryout employment that were not linked to training, 
•infomiation on unemployment rates in the sites is presented in Chapter 7. 



more urban areas are able to provide a more balanced offering of services between OJT and classroom 
training. 

In examining the evidence provided by the 16 study SDAs, there is only weak support for these 
hypotheses. Higher proportions of OJT tend to be somewhat more frequently provided in more rural 
areas, while classroom training typically plays a larger role in more urban areas. For example, two of the 
four most rural areas in the study - Northwest Minnesota and Springfield - placed greater relative 
emphasis on OJT across all target groups. But Butte and Larimer, the other two very rural sites, and 
Marion (which includes several rural counties) are exceptions to this pattern. All five of these SDAs also 
provided some classroom training to ail four target groups, but - with the exception of Marion - 
generally at comparatively low levels. The three largest urban areas in the study - Jersey City, Oakland, 
and Providence - did not consistently provide one service over another across the four target groups, 
although classroom training was, as noted above, more common in urban areas than in rural areas. Fort 
Wayne, another SDA with a large city, did not fit the pattern; in that SDA, on-the-job training and job 
search assistance were strongly emphasized for most target groups. 

As this chapter's discussion makes clear, the National JTPA Study had to be designed to test the 
JTPA system in its diversity, rather than any specific programmatic intervention. This meant that the 
research design had to be flexible while still addressing the key policy questions posed by the U.S. 
Department of Labor. The next chapter explains in more detail the intake process typically used in the 
study sites and the way in which random assignment procedures were superimposed on these existing 
practices. 
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CHAPTER 4 

IMPLEMENTING RANDOM ASSIGNMENT WITHIN THE JTPA PROGRAM 



This chapter presents the steps that potential JTPA enroUees typically followed during program 
intake (labeled "client flow") in the study sites and how the research procedures were superimposed on 
the usual routines J This discussion identifies areas in which implementation of random assignment may 
have affected the normal practices of the site. The chapter concludes by presenting information on one 
key aspect of random assignment: whether the process yielded treatment and control groups that were 
similar on observed baseline characteristics.^ 

This discussion is directly relevant to several key challenges to the validity of the study. First, 
it is important that random assignment be introduced into the normal program intake process with as little 
disruption as possible to the way in which clients typically enter the program. This chapter describes how 
this was done, what special steps were introduced, and the point in the intake process beyond which the 
study was very unlikely to have had any effect on normal operations. Second, random assignment must 
in fact be just that: assignment of sample members to a program (treatment) group or a control group on 
a random basis. While it is not possible to document the unobservable characteristics of the sample 
members and demonstrate that the two research groups are in fact the same on these grr unds, this chapter 
does explain the random assignment process and analyze whether the two groups are similar on observable 
characteristics. 

L Client Flow and Study Procedures 

Exhibit 4.1 presents the basic steps common to all SDAs in the study by which an individual 
eligible for JTPA could apply for the program, be randomly assigned, and (if assigned to the treatment 
group) enroll in the program. Normally, many more people contact the SDA or a service provider to learn 
about the program and inquire about eligibility rules than actually apply.^ Consequently, as Exhibit 4.1 
shows, individuals may "exit" from the intake process at each step. This could occur because they fmd 
other opportunities, decide JTPA is unlikely to provide what they want, or are discouraged by what staff 
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*This material is drawn from Kemple, Doolittle, and Wallace, 1993. 
^This material is based on Appendix A of Bloom et al., 1993. 

^Some SDAs have tracked client contacts and report that about one-quarter of those inquiring about the program 
ever enroll, with much of the drop-off coming early, before eligibility has been determined. 
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EXHIBIT 4.1 

FLOW OF SAMPLE MEMBERS IN THE NATIONAL JTPA STUDY SITES 



jSTAGE 1 1 



ISTAGE 2 



Classroom Training 
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Group 
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Application 



Eligibility Determination 

i 



Infomied Cor^ent Fonn 
and BIF Cof:7ipletion 



Assessment 



Recommendation of Services 
and Service Strategy 



OJT/JSA 
Service Strategy 



Random Assignnr>ent 



Treatment 
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Group 



Enroll 



Do Not 
Enroll 



Receive 




Receive 




Receive 


Services 




Sen/ices 




No Services 



Exit 



Recruitment 








Exit 


* 





Exit 



Exit 



Exit 



Exit 



Other Sen/ices 
Sen/ice Strategy 



Rarxiom Assignment 



Treatment 
Group 



Control 
Group 



NOTE: Because of limited space, the stage 3 sample flow has been fuHy depicted only for the OJT/JSA service strategy, 
although it would be the same for the other two strategies. 
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tell them about their prospects in the program. This section discusses these steps in the client flow shown 
in Exhibit 4.1 by grouping them into three stages. 

A. Stage 1. Connecting with Those Eligible for and Interested in JTPA: 
Recruitment Application, Eligibility Determination, and Background 
Information Form (BIF> Completion 

1. The JTPA>Eligible Population , The study made no change in the eligibility rules for 
the program. The federal JTPA statute defines the basic eligibility requirements for JTPA, which are 
principally low family income during the previous six months or current receipt of public assistance.** 
In each local area served through a single SDA, the size of the eligible population will vary with the 
characteristics of the population and local labor market conditions. However, JTPA is not a legal 
entitlement for all who satisfy the eligibility requirements. Nationally, JTPA funding is sufficient to serve 
fewer than 10 percent of those eligible, so the pool of eligible potential applicants normally greatly 
exceeds program capacity. 

2. Recruitment Nevertheless, recruitment of program applicants takes substantial effort 
in many (though not all) SDAs and can involve both the local administrative entity running the program 
and agencies under contract to provide various kinds of training and employment services. Recruitment 
is a special challenge because SDAs must identify eligible individuals who are interested in the types of 
training and employment services that JTPA provides and are able to participate without receiving stipends 
or other program-related income support.^ For those without other sources of support, the program must 
offer income-generating services such as OJT or job search. 

The experience of program operators indicates that the level of unemployment in the area is the 
best predictor of the difficulty of recruitment: The lower the level of unemployment, the harder it is to 



^JTPA authorizes the provision of job training for economically disadvantaged individuals and others facing 
serious barriers to employment. An "economically disadvantaged individual" is dcfmcd as someone who (1) receives 
cash welfare payments under a federal, state, or local welfare program or is a member of a family that receives these 
payments; (2) has a total family income (excluding unemployment compensation, child support payments, and welfare 
payments) for the six-month period prior to application that, in relation to family size, is not in excess of the higher 
of (a) the poverty standard established by the Office of Management and Budget (OMB) or (b) 70 percent of the 
"lower living" standard, which is the standard above the poverty standard; (3) receives food stamps; (4) is a foster 
child on behalf of whom state or local government payments are made; or (5) as permitted by regulations, is an adult 
individual with a handicap, whose own income meets the family income requirements, exclusive of the income of 
other family members. 

'One of the important changes made in the shift from CETA (the Comprehensive Employment and Training Act) 
to JTPA was that income support (in the form of regular stipends or public service employment) was virtually 
eliminated. Many SDAs dc provide small payments to cover some expenses of participating (e.g., transportation 
expenses), but in general participants must be able to support themselves through nonprogram resources. 
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recruit applicants for JTPA. Because JTPA does not provide in-program income, as jobs become more 
plentiful JTPA becomes relatively less attractive. As evidence of this, even though low-income workers 
are eligible for JTPA, most applicants are unemployed or had previously left the labor force altogether." 

During the period of random assignment, sites had to identify a large enough pool of eligible 
applicants who were interested in participating and "appropriate for the JTPA services" to be able to both 
serve the number of people they wished to enroll and to create a control group. With a random 
assignment ratio of two treatment group members for each control group member, sites had to assure that 
150 people reached the point of random assignment for every 100 individuals they planned to serve. In 
most sites, this could be achieved in two ways. First, site staff could increase recruitment efforts and 
expand their applicant pool, because the eligible population for JTPA remained much larger than the 
number of people recruited in the study SDAs. Second, staff could reduce the "exit" of individuals from 
the applicant process prior to random assignment by streamlining application procedures or marketing the 
potential benefits of JTPA more clearly. The research team, through consultants with expertise on 
itscruitment and applicant retention, offered technical assistance to all SDAs in the study.'' Sites that 
chose to increase recruitment efforts were encouraged by the study team to seek additional applicants who 
fulfilled existing SDA enrollment priorities. 

Because labor maricet conditions changed during the period of random assignment (in most sites, 
unemployment rates declined), it is difficult to isolate the effect of this "study-induced" expansion of the 
applicant pool on the characteristics of applicants. However, program operators in the study sites reported 
that the combination of factors led them to recruit less job-ready applicants than had been the case in the 
mid-1980s, when unemployment was higher. 

Data on the applicant pool in most study sites are not available, and there are no data nationally 
on applicants for JTPA. The JTPA Annual Status Report (JASR) data do include information on the 
characteristics of program "terminees" in each SDA, e.g., the percentage from various racial groups, the 
percentage who were school dropouts, etc. Preliminary analysis using these data suggests that there were 
clear changes in the characteristics of terminees in individual study sites from year to year before and 
during the study. Using information on terminees and local labor market conditions (most importantly, 



*For example, 86 percent of the sample for the National JTPA Study were not working when they applied for 
JTPA. See Bloom, 1991. 

'See Kelly, 1987, and Elsman, 1987, for examples of the assistance offered. These consultants had worked 
widely within the JTPA system prior to the study, and some of the study SDAs were already familiar with their 
work. Technical assistance on recruitment and retention was presented to sites as one of the benefits of participatmg 
in the study, since recruitment was a common problem in JTPA in the late 1980s. 



the unemployment rate) from the JASR data, the researchers looked at key characteristics such as the 
percentage of terminees without a high school diploma or GED and the percentage receiving public 
assistance. One important finding was the extent of year-to-year changes in terminee characteristics in 
SDAs during "normal" (i.e., non-study) years. These were likely to be linked to important programmatic 
changes such as a shift in contractors (e.g., closing out a contractor who drew applicants from one group 
in the community and beginning another contract with a service provider drawing applicants primarily 
from another group') or because of important changes in the applicant pool (e.g., because of a plant 
closing or an influx of new residents). 

In some cases, these changes were consistent with the view that over time the study sites served 
less job-ready applicants as the local unemployment rate declined. But in other sites, there is no clear time 
trend. Further, in some cases the pattern varied depending on which terminee characteristic was examined, 
e.g., the trend for those without a high school diploma might differ from that for welfare recipients. 

In future woric^ when the relevant data on the years following random assignment are available, 
the researchers will compare changes in the characteristics of terminees in the study sites to such changes 
in the other 600-plus SDAs included in the JASR data. The goal will be to see whether there were study- 
induced changes in the characteristics of terminees in the study sites.^ 

3. Application and Eligibility Determination, Those who applied for JTPA continued 
to complete the normal SDA or state forms used to establish their eligibility for the program. They were 
also required to provide the standard documentation (usually information about recent family income or 
receipt of public assistance), the extent of which varied from state to state. SDA staff then had to 
determine if applicants were eligible for the program. The study made no change in the eligibility 
determination process, and staff were told to complete this process as they normally would. Those 
found eligible would continue through to assessment, whereas ineligibles were normally told they could 
not be served." 



•Such changes could be linked to local efforts to meet state or local enrollment goals for specific target groups. 

'Early, very preliminary work on this issue by the nonexperimental researchers on the project did not find any 
pattern of special changes in tcnninec characteristics in the study sites. However, more work is clearly needed 
because of the issues discussed in the text. 

^^As in most studies of ongoing programs, the research design did not include a supplementary check of 
eligibility. This is appropriate because the goal was to measure impacts of the program as it operated normally for 
those on whom the local programs spent funds. 

^^Undcr JTPA rules, up to 10 percent of cnroUecs can have incomes above the normal eligibility cutoff if they 
have other barriers to employment. Deliberately, most SDAs in the study rarely used this exemption, choosing 
instead to use it as a "safety valve" for cases in which auditors found some enroUees ineligible. 
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4. Completiop of the Background Information Form (EIF) . By this point in the process, 
applicants in the study sites typically had completed the study Background Information Form (BIF), with 
assistance from the program staff. The study design allowed some local flexibility as to the precise point 
when the form was to be completed to lessen the burden on local staff who assisted applicants in filling 
it out. In most sites, staff and applicants filled out the BIF when the usual SDA application materials 
w <^re completed. In some instances, completion of the BIF occurred as part of the assessment interviews. 
The BIF provided basic data on the pre-program characteristics of those randomly assigned - information 
that was used to identify members of subgroups analyzed separately in the study. 

B. Stage 2. Identifvine the Research Sample: Assessment Recomm endation for 
Services. Designation of a Service Strategy, and Random Assignment 

1. Assessment Following eligibility determination (or possibly simultaneously for those 
very likely to be eligible), SDA and/or service provider staff assessed the current interests, skills, and 
service needs of applicants. The extent and complexity of this assessment varied greatly among the study 
SDAs; in some it consisted of a multi-day period of testing and interviews, while in others service 
recommendations were based on a short interview at the time the application was completed. Rural areas 
with limited service offerings, in which applicants may have had to travel long distances to the program 
office, were most likely to have combined the application with an abbreviated assessment. 

During the study, local staff were told to follow past practices as closely as possible, and in most 
SDAs assessment practices were not changed by the study. In one SDA, Larimer County, local staff did 
conduct a more thorough initial assessment than had been done in the past in order to make service 
recommendations prior to random assignment. In some SDAs, however, modifications were made because 
of locally initiated changes in program design. For example, in the late 1980s, throughout the JTPA 
system, there was a gradual trend away from multi-step, multi-visit application procedures that were 
commonly used in the early 1980s (when unemployment rates had been much higher) to "screen out" the 
"less motivated" applicants. Several sites in the study (e.g.. Fort Wayne, Omaha, and Corpus Christi) 
made changes along these lines during the period of the study. In addition, changes in contractors in some 
sites led to changes in intake procedures. During the final period of the study, sites also began to test 
reading levels of applicants, under new federal rules requiring reporting of basic skills. 



^^Thcsc data were used in the analysis of the baseline characteristics of the 1 8-month follow-up sample (presented 
in Chapter 7), the comparison of the treatment and control groups (presented later in this chapter), and the analyses 
of subgroups in this and the accompanying impact report. They were also used for the baseline characteristics report 
published earlier in the project (Bloom, 1991). 
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2. Recommendation for Services . Based on their assessment of clients, local staff 
developed recommendations for services reflecting client needs, interests, and preferences. For about 
three-quarters of the 18-month treatment group sample, staff recommended a single service - most 
commonly classroom training in occupational skills (CT-OS) or on-the-job training (OJT), but for the 
remaining one-quarter, staff recommended a combination of services. 

For some applicants, staff decided that no JTPA services would be appropriate and - as would 
normally be the case - these individuals did not continue through the remaining steps of the JTPA intake 
process. This judgment could be made for several reasons: The applicants' skills were too poor to meet 
entrance requirements for JTPA services; staff believed applicants had other problems (such as substance 
abuse or mental health issues or "poor motivation") that posed serious obstacles to success in the program; 
applicants were so employable that the SDA did not want to spend program funds serving them; or the 
SDA did not offer the type of service that the applicants sought." These applicants, whom local staff 
believed to be inappropriate for JTPA, did not become part of the research sample. 

3. Designatiop of a Service Strategy, Since this aspect of the research design was 
complex, some background on its origins is useful. In the planning stage of the study, the U.S. 
Department of Labor set as a goal the estimation of program impacts for important categories of services. 
In addition, in implementing the study, the Department also wished to change local program operations 
as little as possible. Therefore, the research design had to define the study's service categories or 
strategies carefully to reflect actual program practice.'^ A central objective was to develop separate 
categories for individuals recommended for the two most commonly provided, more intensive services 
under JTPA: CT-OS and OJT.*^ But the situation was complicated by the fact that, in a substantial 
minority of cases, local staff typically recommended more than one service. 



*^Thc issue of screening for motivation has been a contentious one in JTPA. Some SDAs have deliberately set 
up application procedures that require people to return several times to complete forms or attend interviews in an 
effort to screen out those without the motivation to persevere. Others argue that such approaches impose many costs 
on applicants (in terms of time) before the benefits of program participation are apparent and result in programs 
losing many applicants who could have been helped. See Kelly, 1987, for one view of this issue. 

'^As discussed later in this section, prior to random assignment, site staff picked recommended services for each 
individual, which implied a service strategy designation in the research. The staff then were expected to follow a 
service plan consistent with this service strategy designation for individuals randomly assigned to the treatment (or 
program) group, although the individual services provided to an individual could vary from their initial 
recommendation. The reason for this approach is discussed briefly later in this chapter and in Doolittle and Traeger, 
1990. Consequently, it was important to develop service strategy deflnitions that would inhibit local discretion as 
to services as little as possible and still produce patterns of post-random assignment service receipt that represented 
real differences in emphasis. 

*^At the time the study started, data from the National Job Training Longitudinal Survey (JTLS) showed that 
about 75 percent of JTPA enrollees nationally were enrolled in only one service and that slightly under 50 percent 
were enrolled in either CT-OS or OJT. During the period of study sample build-up, the proportion of enrollees in 
these two services increased to about 60 percent. 
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The resulting research-defined service strategies, therefore, distinguished between these two 
emphases or primary services, but also accommodated combinations of services. Individuals recommended 
by local staff for CT-OS but not OJT were part of the service strategy labeled classroom training. Those 
recommended for OJT but not CT-OS were part of the service strategy labeled on-the-job training/job 
search assistance (OJT/JSA). This name was chosen because, over the course of random assignment, 
about one-third of the individuals in this category were recommended for both of these services, which 
are designed to get individuals immediately into a job, and a still larger percentage actually received 
JSA.'^ Those recommended for neither CT-OS nor OJT, or for both services, were part of the third 
strategy labeled other services. 

Exhibit 4.2 shows the services under each of the three service strategies used in this and the 
accompanying impact report. Only three restrictions on SDA and service provider actions are implied. 

Restriction #1. Individuals designated for the classroom training service strategy (i.e., 
recommended for CT-OS as their primary service) normally were not to receive OJT, and 
those recommended for the OJT/JSA service strategy normally were not to receive CT- 
OS. 

This restriction seemed consistent with operational reality. SDA experience suggested that clients 
tend to divide into (1) those who are interested in and able (in terms of educational attainment, learning 
style and study habits, and a source of income while in training) to participate in classroom-based 
occupational training, and (2) those who are seeking immediate employment because of financial 
obligations or an aversion to the classroom setting but who also need training. Thus, these service 
strategies accommodated the needs and interests of most JTPA applicants, and client preferences were an 
important factor in assessments in most SDAs. In addition, SDAs rarely provided both CT-OS and OJT 
to a single client, since the combined cost of these two services exceeded what most JTPA programs were 
prepared to spend on one person. 

As a fiirther effort to avoid distorting service delivery in unusual cases, part-way through the 
study, SDAs were allowed to provide a small amount of OJT following CT-OS in cases where it was 
needed to secure placement in a job. This OJT could not amount to more than 20 percent of total training 
time. Similarly, SDAs were allowed to provide a small amount of CT-OS (again, no more than 20 
percent of the training time) when needed to give the applicant a "skills brush-up" in order to secure an 



^*This last point is discussed in Chapter 5. 



EXHBIT 4.2 



PROGRAM SERVICES AVAILABLE IN THE THREE 
SERVICE STRATEGIES 



Classroom Other 
Program Service Training OJT/JSA Services 



Classroom Training in 
Occupational Skills (CT-OS) 


Yes 


No (a) 


Yes 


On-the-Job Training (OJT) 


No (a) 


Yes 


Yes 


Job Search Assistance (JSA) 


Yes 


Yes 


Yes 


Basic Education (BE) 


Yes 


Yes 


Yes 


Miscellaneous Services (b) 


Yes 


Yes 


Yes 



NOTES: (a) In general, this program service was not available, but, as discussed in the 
text, there were exceptions. In certain circumstances, a short CT-OS "brush-up" course 
was allowed prior to OJT in the OJT/JSA service strategy, and brief OJT could follow 
CT-OS in the classroom training service strategy. 

(b) Miscellaneous program services include such activities as assessment, job- 
readiness preparaticm, vocational exploraticm, job shadowing, and tryout employment. In 
this table, miscellaneous program services also include work experience, which is 
identified as a separate program service in other reports on the National JTPA Study. 
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OJT position. As the sen'ice enrollment data presented later in this report show, this option was used vejy 
rarely. 

Restriction #2, There was a ceiling on the proportion of clients who could be designated 
for the "other services" strategy. 

Since this strategy contained no restrictions on subsequent service rvceipt, SDAs would have had 
an incentive to designate everyone for it. Consequently, the research team negotiated with each site a 
ceiling on the strategy, with the level reflecting past patterns of service receipt. For example, in SDAs 
where most individuals typically received only OJT or CT-OS, the ceiling on other services was set low; 
in SDAs where many individuals received a combination of CT-OS and OJT, or services other than these 
two, the ceiling was set much higher. Among the 16 SDAs in the study, the range was from 15 percent 
to 70 percent, with more than half the sites at 35 percent or lower. Over the course of random 
assignment, most sites recommended many fewer applicants for the other services strategy than their 
ceiling allowed. Overall, the proportion of the 18-month treatment group sample in this strategy was 27 
percent. 

Restriction #3. For applicants randomly assigned to the treatment group, site staff had 
to put in place a service plan in which the individual services were consistent with the 
designated service strategy. 

While the service strategy categories were flexibly defined, site staff were expected to provide 
services consistent with these definitions. Over the course of working with clients, site staff could provide 
individual services different from those originally recommended, but the services they did provide were 
expected to be consistent with the service strategy originally chosen.^^ 

In some unknown (but probably small) percentage of treatment group members, this restriction 
did prevent staff from making changes in a service plan they might otherwise have made. The belief that 
this percentage was small rests on two features of the implementation of the study. First, site staff were 
included in the discussions about service strategy definitions prior to the start of random assignment and 
believed they were consistent with their practices in the vast majority of cases. Second, sites participated 
voluntarily. Had the service strategy definitions proved overly restrictive, sites would have raised this as 
a serious issue following the start of random assignment. Over the course of random assignment, very 



*'Such changes in services were most common in the OJT/JSA service strategy, where nearly as many cnrollees 
received only JSA as received OJT, even though JSA was originally recommended for only one-third of the group. 
Sec Chapter 5 for the details. 
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few complaints were voiced by local staff despite the fact that the service plans for more than 14,000 
people were subject to these restrictions. 

4, Random Assignment to the Treatment or Control Group . The random assignment 
procedure was straightforward. Site staff called MDRC (using a toll-free number), provided background 
information on each individual to allow tracking of sample build-up by target group, and listed the service 
recommendation and service strategy designation. MDRC staff gave site staff the person's research status 
(treatment or control). Site staff were then to follow their service recommendations for those randomly 
assigned to the treatment group. MDRC staff kept a roster of those randomly assigned and their research 
status to assure that if a person in the sample reapplied to the program, he or she would be treated 
consistently during the follow-up period. 

Those in the control group were excluded from JTPA-fiinded services in the SDA for the 
following 18 months, but were provided with a list of alternative service providers in the community 
whom they could contact on their own.'* Exhibit 4.3 shows that sites were quite successful in enforcing 
the "control embargo" rule. About 3 percent of the controls were enrolled in any JTPA services at any 
point during the follow-up period.*^ 

Essentially, as shown in Exhibit 4. 1, this process created a separate control group for each service 
strategy, permitting separate impact estimates for how each service strategy "worked" for the people 
recommended for it. As mentioned earlier, the test was of the impact of the added services received by 
the treatment group in each service strategy above the level received by the comparable control group - 
an "incremental" rather than a "no-service" comparison. The process also permitted a combined impact 
estimate for the entire sample. 



**After 18 months, members of the contiol group could receive JTPA services if they returned to the SDA or 
service provider on their own. However, SDA staff agreed not to make any special effort to recruit members of the 
control group at the end of their exclusion from JTPA. 

^^In fact, the actual level of improper "crossovers" from the control group to JTPA services was lower than this. 
Some members of the research sample were participatmg m a title of JTPA not included in the study (e.g., the 
summer youth program) prior to random assignment and had applied for Title Il-A services. When they applied for 
and were judged appropriate for these Title 11-A services, they became part of the study and were subject to random 
assignment. Those assigned to the treatment group were given access to Title-ll-A services, but those assigned to 
the control group were not. However, since control group members were allowed to finish their non-Title-Il-A 
services, they would appear as enrolled in JTPA at the beginning of the follow-up period. Of the 159 members of 
the control group with any JTPA enrollment during the follow-up period, 60 began their enrollment before random 
assignment, mostly in non-Title-Il-A services. Thus, less than 2 percent of controls began a JTPA enrollment after 
random assignment. Youths were the most common group randomly assigned when already enrolled in another title 
of JTPA (the summer youth program), which explains the higher rates of control group enrollment for the youth 
subgroups in Exhibit 4.3. 
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EXHIBIT 4.3 



ENROLLMENT RATES FOR CONTROL GROUP MEMBERS, 
BY SERVICE STRATEGY AND TARGET GROUP 



Service^ Strategy 


All Target 
\jroups 

(%) 


Adult 
Nien 

(%) 


Adult 
(%) 


Male 

Youths 

(%) 


Femaie 
Youths 
(%) 


Classroom Training 


3.2 


2.6 


2.6 


6.6 


3.3 


OJT/JSA 


2.4 


1.5 


2.8 


3.6 


3.7 


Other Services 


3.1 


2.5 


3.3 


2.0 


5.1 


All Strategies 


2.9 


2.0 


2.8 


3.9 


3.9 


SOURCE: MDRC calculations from Background Information Form responses and program 
enrollment data from the 16 SDAs. 

NOTE: Calculations for this table are based on data for all 5,556 control group members 



in the 18-month study sample. 



Local staff recommended people with different characteristics for the three service strategies, so 
at random assignment the sample in each strategy differed in important ways from the others. These 
differences translated into different levels of post-random assignment control group earnings across the 
three service strategies and four target groups, as shown in Exhibit 4.4. For example, adult women 
recommended for the classroom training service strategy who were randomly assigned to the control group 
had average earnings of $6,391 during the 18-month follow-up period.^° For adult women and both 
youth target groups, the mean earnings of the OJT/JSA service strategy control group were the highest, 
suggesting that those recommended for this strategy were - as a group - more employable than the rest 
of the sample. For adult men, the control group for the other services strategy had slightly higher earnings 
because — more so than for the other target groups - these men were employable enough that staff 
recommended for them only job search assistance. Because of these baseline differences in characteristics 
and resulting differences in control group earnings, it is not possible to estimate experimental comparisons 
of the effectiveness of the three service strategies. The treatment and control groups in each strategy 
differed in other characteristics besides the type of service recommended by staff, making it difficult to 
isolate the effect of the ^ vice strategy on impacts.^' 

C. Stage 3. Efforts to Arrange Services for the Treatment Group: Enrollment in 
JTPA and Involvement of the NonenroUed with JTPA 

Individuals randomly assigned to the treatment group were offered access to JTPA services. This 
offer and the resulting services provided were the "treatment" being tested through the random assignment 
experiment. In effect, this design was testing the impact of the decisions that local staff and individual 
applicants made during the intake process: whether to tiy to arrange and participate in JTPA services. As 
will become clear in the following discussion, many factors affected whether an individual actually ended 
up enrolled in JTPA. Also, arranging services involved considerable administrative effort and expended 
program resources; thus, starting the impact story at the point of referral to services makes sense. 

The impact analysis in the National JTPA Study presents impact estimates for all persons assigned 
to the treatment group and - alternatively - for all those in the treatment group who participated in 
JTPA.^^ However, as this section explains, the concept of "participation" in JTPA is somewhat vague 
and hard to measure, complicating the interpretation of these alternative impact estimates. 




^^ese estimates of average earnings for the control group includes zero earnings for individuals who did not 
work. 

^^This issue was discussed in Chapter 2. 

^Sce Bloom ct a!., 1993, for a discussion of this, 
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EXHIBIT 4.4 



MEAN EARNINGS FOR CONTROL GROUP MEMBERS 
DURING THE 18-MONTH FOLLOW-UP PERIOD, 
BY SERVICE STRATEGY AND TARGET GROUP 





Adult 


Adult 


Female 


Male 




Women 


Men 


Youths 


Youths 


ocrvice oiraiegy 










Classroom TraiDing 


6,391 


11,780 


5,936 


,783 


OJT/JSA 


8,607 


12,456 


7,620 


12,765 


Other Services 


7,960 


12,516 


5,726 


9,839 


All Strategies 


7,488 


12,306 


6,225 


10,736 


Sample Size 


2,142 


1,867 


835 


708 



SOURCE: Adapted from Bloom et al. , 1993; estimates are based on First Follow-up Survey responses. 
The estimates for adult women are also based on earnings data from state unemployment 
insurance (UI) agencies. 



NOTE: Estimates are regression-adjusted to control for differences in baseline characteristics between 
the treatment group and control group. 



1. EnroHment in JTPA Services , Enrollment in JTPA occurs when SDA staff enter a 
person*s name and application data into the local JTPA management infonnation system (MIS) and enroll 
her or him in one or more specific JTPA-funded services. This step makes the person an official JTPA 
participant, whose service receipt and progress are tracked and whose termination and post-program status 
(e.g., employment and wages) are noted as part of the JTPA performance standard system,^^ By 
enrolling clients, SDA staff are held accountable through the JTPA performance standard system for the 
costs that JTPA incurred in serving them and for their success when they left the program,^* As 
discussed in more detail in Chapter 5, 64 percent of the 18-month treatment group sample were enrolled 
in JTPA at some point in the 18-month follow-up period. Thus, Exhibit 4,1 shows individuals who 
enrolled in JTPA as a subset of each of the three treatment groups. 

2. Nonenrolled Treatment Group Members , Four factors help explain why some 
members of the treatment group would never be enrolled in JTPA. 

a. Despite the initial assessment that a client is appropriate 
for JTPA, staff may be unable to find a service provider 
willing to accept the person. Service providers often have 
the final say on whether they will accept an applicant. 
Many classroom training agencies have entrance 

^ requirements as to minimum initial skills, and employers 

^ who can provide OJT will typically make their own 

assessment about whether they want the ^plicant as an 
employee. In addition, an applicant may be seeking 
classroom or other training at a time of the year when it is 
not offered. 

b. Applicants may change their minds about JTPA as they 
continue to seek other opportunities or learn more about the 
program. Many are looking for work on their own, and 
some will find employment. Some may discover different 
ways to finance the type of training they seek. And still 



"The 1992 amendments to JTPA and subsequent regulations issued by the Department for program year 1993 
made major changes in this aspect of the program, 

"As discussed in Chapter 1, during the period of the study, clients' employment was measured as of their point 
of termination from the program. Starting in program year 1990, performance standards were amended to include 
measures of employment 13 weeks after leaving the program. Controversy over the costs standards led the 
Department to end the use of the cost standards in program year 1990, One major concern was whether a focus on 
costs per placement for adults or per positive tennination for youths provided too many incentives to nm short-term, 
low-intensity programs, A second concern was that the focus on JTPA program costs could provide a distorted cross- 
site comparison, since local programs varied in their ability to draw on other funders to provide services for JTPA 
clients. 
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others may decide that they are not interested in a job or 
training after all. 

c. The design of the JTPA program encourages local staff 
to make sure that applicants are going to participate and 
do well in a service before they are enrolled and counted 
as a JTPA participant. The high visibility of standards 
within the JTPA system goes well beyond their limited role 
in allocating incentive grants, consisting of up to 6 percent 
of Title ll-A fiinds. An SDA's success in exceeding its 
performance standards is often seen as a sign of how well 
a program is operating. This clearly encourages SDAs to 
focus on achieving their standards on the various outcomes 
measured, but it also creates an incentive to hold off on 
enrolling individuals (i.e., having them count as part of the 
program) until they are placed in and begin a service that 
staff feel is likely to produce success." 

d« Many SDAs have believed that they have discretion in 
defining the point at which individuak "count" in their 
performance standards and have responded to the 
incentives of the standards by delaying enrollment. In the 
initial years of JTPA, the Department adopted the position 
that JTPA was to be primarily controlled by states and 
localities. Therefore, it did not define precisely many key 
administrative terms, including the point at which 
enrollment should occur. In recent years, this federal stance 
has changed, but a holdover from the initial period is the 
continued practice of linking enrollment to the actual 
beginning of the intended service, be it the first day of class 
attended, the first workday for OJT, or (in an extreme 
example) when participation in job club or job search 
assistance leads to employment. 

There is no national survey that has collected information on the point of enrollment in local 
JTPA programs. However, most of the study sites enrolled individuals in classroom training when they 
attended their first class or in OJT when they worked their first day, although one site did enroll people 
during assessment. In a few sites, local staff could refer people to job search assistance or job club 



Although the perfomancc standards system has changed in recent years to include longer-term measures of 
success (i.e., 13 weeks after leaving the program) and measures less closely tied to immediate employment and low 
costs, the basic incentives remain unchanged. SDAs can be regarded as well run. and gain some additional funding, 
if a high proportion of people leaving their program find a job that pays well or have attained a variety of 
employability-enhancing competencies. 
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without enrolling them and observe how they acted in this setting as part of an "extended assessment." 
The applicants may never have been enrolled in JTPA unless they found a job or, because their behavior 
in the job club showed motivation and promise of employability, were referred to another service.^^ 

These factors suggest that nonenroUed members of the treatment group could have had some post- 
random assignment involvement with the JTPA system. This could complicate the calculation of impacts 
per person "participating" in JTPA because enrollment and receipt of some JTPA assistance may not be 
synonymous. In order to understand the extent to which this occurred, the research team drew a sample 
of nonenroUed treatment group members in 12 sites^^ and talked with local staff about their efforts to 
work with these individuals after random assignment. 

Exhibit 4.5 presents the findings from this special study. The local staff had no contact with 15 
percent of this sample of nonenroUees after random assignment; basically, they were unable to locate them 
again. Another 1 1 percent reported that they were no longer interested in JTPA, for a variety of reasons. 
Another 20 percent of these nonenroUees were reconticted, but staff never arranged service for them. The 
remaining 53 percent of the sample of nonenroUed treatment group members had some post-random 
assignment involvement with JTPA without being enrolled. The most common service, provided for 36 
percent of the nonenroUee sample, was one or more referrals to employers for a possible OJT position. 
Twenty percent participated in job club or other job search assistance. This small study suggests that local 
staff worked with about half of the treatment group members who never enrolled in JTPA, though in 
many cases little service was provided. 

These results suggest that local staff worked (with a wide range of intensity and commitment) with 
about four-fifths of the treatment group. This consisted of the 64 percent who were enrolled, plus an 
additional 18 percent (i.e., half of the 36 percent of nonenroUees) who were never enrolled but did have 
some post-random assignment JTPA involvement.^' 



^^Legislation enacted in 1992 tightens some of the definitions of key administrative terms in recognition of this 
problem. In addition, the Department has recently issued new guidelines defining more precisely key administrative 
terms. 

^'Thc samples were drawn in two time periods: November 1988-January 1989 and March-June 1989, and 
included most of the SDAs where random assignment was occurring at those times. 

"This finding docs not relate directly to the ongoing debate about how much local JTPA programs "cream" in 
selecting cnroUces, i.e., seek more employable clients and shy away from enrolling hardcr-to-scrvc clients. Much 
of this creaming, if it existed, could have occurred before the point of random assignment, the point at which the 
18-month treatment group sample was identified. Random assignment followed local staffs assessment of clients 
and a decision that they would be appropriate for the program. 
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EXHIBIT 4.5 



PERCENTAGE DISTRIBUTION OF 
POST-RANDOM ASSIGNMENT ACTIVITY IN JTPA 
OF TREATMENT GROUP MEMBERS WHO DID NOT ENROLL 





Nonenrollees 


Activity 


(%) 


fMu A uruicr v-Aiuuivfc 


15 


Further CooUct, But Not Eligible 


1 


No Longer Interested (a) 


11 


Got job on own 


5 


Moved 


2 


Health problems 


1 


In another program 


1 


Reason imknown 


3 


Interested, But Made ConUct Only 




mjyj Received No Services 


20 


Interested and Received Service(s) (b) 


53 


Received further assessment 




and counseling 


11 


Referred to classroom 




training provider(s) 


5 


Received support service(s) 


2 


Referred to employer(s) for 




possible on-the-job training 


36 


Participated in job club or 




received job search assistance 


20 


Toul 


100 


Sample Size 


307 



SOURCE: Information collected by MDRC site representatives during the National 
JTPA Study. 



NOTES: Calculations for this table are based on data for a random sample of 
307 treatment group members in the 18-month study sample who did not enroll in 
JTPA. 

(a) When totaled, the subcategory percentages are over 11 percent because 
nonenrollees could cite more than one reason for no longer being interested in JTPA. 

(b) When totaled, the subcategory percentages are over S3 percent because 
some nonenrollees received more than one service. 



n, A Comparison of the Treatment and Control Groups 



In a properly designed and executed random assignment process with large samples, the treatment 
and control groups are expected to be very similar in their measured baseline characteristics. This section 
summarizes a detailed analysis presented in other reports, which found that in fact the treatment and 
control groups in this study were very similar for each of the four target groups.^' 

The Background Information Form collected many baseline characteristics on sample members, 
allowing a detailed look at this issue. Rather than examining the size or frequency of differences between 
the control group and the treatment group on individual characteristics, the research team rested its 
conclusion that the two groups were similar on a multivariate discriminant analysis.^*^ Using this 
approach, the research team was able to test the hypothesis that the baseline characteristics do not 
discriminate between the two populations; in other words, that the mean values for the two populations 
on the characteristics are the same. 

Using this approach to pool the comparisons of all the individual baseline characteristics into one 
joint test per target group. Exhibit 4.6 presents the ''incremental R^" for the baseline characteristics for 
each group, where this is defined as the difference between the R^ for the full model for the group and 
the R^ for the model that omitted the baseline characteristics. The very low incremental R^ values in the 
exhibit indicate that the addition of the baseline characteristics adds very little to the power of the model 
to discriminate between the treatment and control groups. In other words, it indicates that there is 
virtually no correlation between whether a person was a member of the treatment or control group and 
his or her measured baseline characteristics. The p-value for the incremental R^ indicates the chance of 
the observed differences in characteristics between the treatment and control group samples occurring if 



^^is section is based primarily on Appendix A to Bloom ct al., 1993, which analyses the 18-month study 
sample. Bloom, 1991, contains a similar analysis and findings for the full experimental sample. 

choice of a multivariate analysis was made for three reasons: (1) to avoid a multiple comparisons problem, 
in which, if enough comparisons of individual characteristics of two samples are made, some comparisons will show 
differences, even if the populations from which the samples arc drawn are identical; (2) to account for the 
interdependency of some baseline characteristics (e.g., education and earnings); and (3) to permit analysis of 
individuals when some data on individual characteristics were missing. See Bloom et al., 1993, Appendix A, for 
a full discussion of the analysis methodology. This approach reJies on an assumption that the characteristics of the 
population from which the treatment and control groups are drawn have a joint normal distribution, with a common 
covariance matrix but different means. It rests on an ordinary least squares regression of the following fomi: 
Treatment status is a function of an intercept temi, the value of the baseline characteristics, and a random error term. 
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EXHIBIT 4.6 



RESULTS OF THE MULTIVARIATE DISCRIMINANT ANALYSIS 
OF BASELINE CHARACTERISTICS OF THE TREATMENT 
GROUP AND CONTROL GROUP, BY TARGET GROUP 





Aduit 
Women 


Adult 
Men 


Female 
Youths 


Male 
Youths 


Sample Size 


6.607 


5,626 


2,649 


2,144 


2 

Incremental R 


0.0032 


0.0065 


0.0128 


0.0100 


p- Value for the 

2 

Incremental R 


0.95 


0,29 


0.40 


0.94 



SOURCE: Bloom ct al., 1993. 



GO 



there was in fact no difference on average in the population from which the samples were drawn.^* This 
probability ranges from 29 to 95 percent across the four target groups. Under this test, the usual standards 
of statistical significance would reject the hypothesis of a difference in measured characteristics if the p- 
value exceeds .1. Thus, this test presents strong evidence for their similarity. 

In summary, the findings of this analysis indicate virtually no difference in measured baseline 
characteristics of treatment and control group members. Those differences that were observed are neither 
substantial nor statistically significant. 



''This was calculated by computing the F statistic for each incremental and determining the probability of 
an F value being equal to or greater than the observed value under the null hypothesis of no difference between the 
baseline characteristics of the treatment and control groups. 
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CHAPTERS 

SERVICE RECEIPT BY THE TREATMENT AND CONTROL GROUPS 



The National JTPA Study compares the employment, earnings, and other outcomes for the 
treatment group — which could receive employment and training services from JTPA and other sources 
- with those for the control group - which had access to non-JTPA services. This chapter outlines the 
extent and type of JTPA services in which members of the treatment group were enrolled, in the process 
explaining differences among the service strategies and key subgroups J It then discusses service receipt 
by members of the control group and compares the service receipt of the two research groups (whether 
from JTPA or other programs),^ This information on the service difference between the treatment and 
control groups is vital background for interpreting the impacts presented in the accompanying 18-month 
impact report. 

It is important to keep in mind the goals of the National JTPA Study as a context for this 
discussion of the services received by the treatment and control groups, and to compare these with the 
goals of a demonstration of a special program. The comparison focuses on two implications of the 
differing goals: the extent to which there is unifomiity in the services provided to individuals and the 
likely service difference between the experimental and control groups. 

The National JTPA Study was undertaken to assess the effectiveness of JTPA as it normally 
operates in a diverse group of local programs. Consequently, because there is great variety in the JTPA 
services provided to individuals, this study is not testing any particular program model. The service 
strategies for which impacts are estimated, as discussed in Chapter 4, are intended to group together 
individuals for whom local staff recommended similar types of services, but even within these groupings 
there turns out to be considerable diversity in services. Furthermore, in an ongoing program such as 
JTPA, the service difference between the treatment and control groups is hard to predict. As Chapter 4 
discussed, in a typical program some of those in the treatment group will not be served at all, and still 
more will not be formally enrolled in JTPA. Also, some in the control group will find services elsewhere. 
As mentioned in Chapter 2, for example, in some study sites most controls were likely to receive job 



*This is discussed in more detail in Kcmplc, Doolittle, and Wallace, 1993, and in Bloom ct al., 1993, Chapter 

3. 

^In the process, it briefly discusses issues encountered in the effort to measure non-JTPA services through the 
18-month follow-up survey. This information is discussed in more detail in Bloom ct al., 1993, Chapters 4-6. 
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placement leads from state Employment Service staff who were co-located with JTPA. If there is no 
difference in the type, level, and quality of services received by the treatment and control groups, 
experimental estimates will show no zero impacts, but this does not mean that the services made no 
difference as compared to an alternative of no services. However, it would undercut the goal of studying 
how JTPA normally operates in a diverse group of local programs if special measures were taken :n site 
selection or in implementation of the study to assure that the treatment group participated an extraordinary 
amount in JTPA or the control group participated very little in alternative services. Thus, there may be 
a tradeoff between the goals of studying JTPA as it operates normally and learning the maximum amount 
about the effectiveness of specific types of services.^ 

In a demonstration of an innovative program, in contrast, sites are typically providing a common 
set of cervices, though there is inevitably some local variation in how the program is actually 
implemented. In many demonstrations, the organizers provide special assistance (financial and otherwise) 
to help sites implement the program model in the best way. Further, the target group for the program (and 
to some extent the sites) may be selected with the goal of addressing the needs of a previously 
underserved group. These features have the ancillary benefit of maximizing the service differential 
between the treatment and control groups. 

Throughout the National JTPA Study, the researchers faced the challenge of striking the best 
balance between studying JTPA as it normally operates and learning the most possible about the 
effectiveness of services. This chapter's summary of the services received by the members of the 
treatment and control groups provides the reader with the background to understand how this tension was 
resolved in the study and provides information vital to the proper interpretation of the impact findings. 
The picture the chapter paints is of a modest though real service difference between the treatment and 
control groups. This finding is important in interpreting the meaning of the impact estimates reported in 
the accompanying report; positive impacts of a given size arc more encouraging if they are the product 
of a modest difference in services between the treatment and control groups than if they are for a fully 
and intensely served treatment group and a no-service control group. 



'There is a related tradeoff in the noncxpcrimcntal analysis of program impacts that prevents that type of analysis 
from being a simple way to solve this problem. The more successful researchers arc in identifying a comparison 
group that receives none of Ihe services under study, the more likely the members of this comparison group arc to 
differ from program participants in important, but difficult to measure, ways. 
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JTPA Service Enrollment by the Treatment Group 



X. Service Recommendations 

The starting point for analyzing service enrollment by the members of the treatment group is the 
recommendations for individual services and a service strategy made by local staff prior to random 
assignment. These represent the intended services for the individuals in the sample. 

Exhibit 5.1 shows the service recommendations made by local staff for the entire 18-month 
treatment group sample, and for individuals grouped into the three service strategies defined for this study: 
classroom training, OJT/JSA, and other services. For all groups (shown in the first column), staff 
recommended the classroom training service strategy for 36 percent of the individuals, OJT/JSA for 37 
percent, and other services for 27 percent. Under each service strategy subheading, the rows show the 
percentage of the sample that staff recommended for the individual services and combinations of 
services.* Over 75 percent of the entire sample was recommended for a single service. 

The pattern of service recommendations varied greatly among the four target groups shown in the 
remaining columns of Exhibit 5.1. Staff recommended the classroom training service strategy for 44 
percent of aduk women and female youths compared to only 25 percent of adult men and 30 percent of 
male youths. Staff recommended the OJT/JSA service strategy for 49 percent of aduk men and 33 percent 
of male youths compared to 35 percent of aduh women and 23 percent of female youths.^ 

B. JTPA Enrollment 

Exhibit 5.2 presents JTPA enrollment rates for all treatment group members and rates 
disaggregated by service strategy and by target group.* Sixty-four percent of the entire treatment group 
were enrolled in JTPA at some point in the 18-month follow-up period covered by this report, as shown 
in the bottom row ("All Strategies'') of the column headed "All Target Groups." Classroom training had 
the highest enrollment rate (72 percent) and OJT/JSA the lowest (57 percent) for all target groups.' 
Within each service strategy, the variation in JTPA enrollment rates across the target groups was relatively 



^For example, staff recommended 31.5 percent of all groups for CT-OS as a sole service, 1 percent for CT-OS 
and job search assistance, etc. 

^Members of the nonexpcrimental analysis team of the National JTPA Study are seeking ways to model 
statistically the process of assessment and service recommendation in 4 of the 16 study SDAs. Sec Bloom et al., 
1990, Chapter 6, for a discussion of the plans for this research. 

*The information presented in this seaion is based on the JTPA enrollment, participation, and temiination data 
prepared by the study sites as part of the nomial program pcrfomiance standards and reporting requirements. 

'A later section of this chapter discusses possible reasons for this pattern. 



EXHIBIT 5.1 



PERCENTAGE DISTRIBUTION OF SERVICE STRATEGY RECOMMENDATIONS 
AND RECOMMENDED PROGRAM SERVICES, 
BY TARGET GROUP 





All Target 


Adult 


Adult 


Male 


Female 


Recommended Service Strategy 


Groups 


Men 


Women 


Youths 


Youths 


and Program Services 












Classroom Training 


35.9 


24.6 


44.0 


29.9 


44.3 


CT-OS only 


31.5 


21.4 


38.6 


26.4 


39.3 


CT-OS and JSA 


0.8 


0.6 


1.3 


0.2 


0.7 


CT-OS and BE 


0.6 


0.2 


0.5 


1.2 


1.2 


CT-OS and misc. services (a) 


0.2 


0.1 


0.2 


0.3 


0.5 


CT-OS and 2 or more 












othftr SArvicAS 


2.7 


2.4 


3.4 


1.7 


2.6 


OJT/JSA 


37.4 


48.7 


35.0 


32.9 


23.2 


OJT only 


24.0 


29,8 


23.6 


19.9 


16.0 


OJT and JSA 


8.1 


12.2 


7.0 


6.5 


3.7 


OJT and BE 


0.2 


0.2 


0.0 


0.5 


0.1 


OJT and misc. services (a) 


0.6 


0.1 


1.1 


0.6 


0.7 


OJT and 2 or more 












other services 


4.5 


6.5 


3.2 


5.4 


2.7 


Other Services 


26.7 


26,7 


21.0 


37.3 


32.5 


JSA only 


5.8 


8.6 


5.2 


4.9 


2.0 


BE only 


2.7 


0.4 


1.1 


8.5 


6.9 


Misc. services only (a) 


13.2 


14.4 


10.0 


16.5 


15.7 


CT-OS and OJT 


0.3 


0.1 


0.4 


0.3 


0.3 


JSA and BE 


0.1 


0.1 


0.0 


0.1 


0.1 


JSA and misc. services (a) 


1.6 


1.9 


1.6 


1.4 


1.3 


BE and misc. services (a) 


1.1 


0.2 


0.3 


3.6 


3.3 


3 or more services 


2.0 


1.0 


2.4 


1.9 


2.9 


Total 


100.0 


100.0 


100.0 


100.0 


100.0 


Sample Size 


11,474 


3,759 


4,465 


1,436 


1,814 



SOURCE: MDRC calculations from Background Information Form responses. 



NOTES: Calculations for this table are based on daU for all treatment group members 
in the 18-month study sample. 

Recommended service strategy distributions may not total 100.0 percent 
because of rounding. Recommended program service distributions may not total the percentage 
reconmiended for the service strategy because of rounding. 

(a) Miscellaneous program services include such activities as assessment, 
job-readiness preparation, vocational exploration, job shadowing, and tryout employment. 
In this table, miscellaneous program services also include work experience, which is 
identified as a separate program service in other reports on the National JTPA Study. 



EXHIBIT 5.2 

ENROLLMENT RATES FOR TREATMENT GROUP MEMBERS, 
BY SERVICE STRATEGY AND TARGET GROUP 



Service Strategy 


All iarget 
Groups 

(%) 


AuUll 

Men 

(%) 


Women 

(%) 


iViaie 
Youths 

(%) 


Pemale 
Youths 
(%) 


Classroom Training 


72.4 


71.2 


72.8 


74.8 


71.5 


OJT/JSA 


56.5 


56.6 


55.4 


58.5 


57.5 


Other Services 


62.3 


58.9 


62.4 


67.7 


63.1 


Ail Strategies 


63.8 


60.8 


64.6 


66.8 


65.5 



SOURCE: MDRC calculations from Background Information Form responses and program 
enrollment data from the 16 SDAs. 

NOTE: Calculations for this table are based on data for all treatment group members in the 
18-month study sample. 



small. Aggregating across all service strategies, adult men were slightly less likely to enroll than were 
the other target groups. 

JTPA Enrollment in Specific Services 

Exhibit 5.3 presents details on the nature of the JTPA services in which treatment group members 
in the entire sample, each target group, and each service strategy were enrolled.* The first column 
presents information for all target groups combined, and the remaining columns list the four target groups 
separately. The top panel is for all service strategies combined, with the remaining three panels presenting 
information for the three service strategies. The rows within each panel show the percentage of the 
relevant group who were never enrolled in JTPA and who were enrolled in specific services. 

For the entire sample (the "All Target Groups" column), 36 percent were never enrolled in JTPA. 
Their most common JTPA service was classroom training in occupational skills (23.9 percent), followed 
closely by job search assistance (23.0 percent). Thirteen percent of the sample were enrolled in on-the-job 
training, with 14.1 percent enrolled in a variety of miscellaneous JTPA services. Ten percent of the 
sample were enrolled in basic education witliin JTPA. However, these summary figures hide considerable 
variety among the service strategies and target groups, as shown in the remaining panels of the exhibit. 

1. C lassroom Training Service Strategy ♦ As shown in the first column of the second panel, 
28 percent of this subgroup were never enrolled in any JTPA service during the follow-up period, with 
only small variation among the target groups. Among the subgroup as a whole, classroom training in 
occupational skills was by far the most common JTPA service (56 percent of the entire subgroup), and 
there was little variation in the percentage enrolling in it among the four target groups. Job search 
assistance, in many cases provided as a follow-up service, was the second most common service. This 
service was much more common for youths (who appeared to need more placement assistance following 
training) than for adults. Basic education was the third most common service among those recommended 
for the classroom training service strategy, and again it was more common for youths than adults. As 
intended, a very small percentage (3.8 percent) of this subgroup were enrolled in on-the-job training. 

2. OJT/JSA Service Strategy^ As previously mentioned, the JTPA enrollment rate was lowest 
for this subgroup, with nearly 44 percent never enrolled in JTPA during the follow-up period and little 



•it shows the percentages of treatment group members who were enrolled in various JTPA services. Control 
group members are not included in this analysis. Since enrollees could receive more than one of the listed services, 
the percentages for enrollment in the individual services in the top panel add to more than the overall enrollment 
rate in JTPA. 
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EXHIBIT 5.3 



SERVICE RECEIPT RATES FOR TREATMENT GROUP MEMBERS, 
BY SERVICE STRATEGY AND TARGET GROUP 



All Target Adult Adult Male Female 
Service Strategy and Groups Men Women Youths Youths 
Program Service (%) (%) (%) (%) (%) 

All Strategies 



Never Enrolled 


36.2 


39.2 


35.4 


33.2 


34.5 


Classroom Training in 
i^ci4)auonai oKiiis ^ci^Lio) 




16.0 


30.5 


18.7 


28.2 


i>asic cuucauon [josl) 


10.0 


5.5 


7.9 


17.7 


18.2 


\Jii IOC J UD 1 lallllll}^ / 




ICC 


lz.6 


12.8 


9.4 


JOD ocarcn y\ssistance 




Z4.4 


Zl.7 


24.2 


22.6 


WI /"Wv T» V no t ■ o ffi r* o 

▼vuiK. HApcnencc 




1 o 
l.o 


3.Z 


4.5 


4.9 


Misceuancous oervices ^a^ 


1i4 1 

14. 1 


IJ.J 


13.6 


17.8 


14.3 


Sample Size 


11,474 


3 J 59 


4,465 


1,436 


1,814 


Classroom Training 












Never Enrolled 


27.6 


28.8 


21.2 


25.2 


28.5 


Classroom Training in 
Occupational Skills (CT~OS) 


56.2 


55.7 


Sl.i 


52.4 


54.8 


Basic Education (BE) 


12.9 


8.8 


10.6 


23.3 


17.8 


On-the-Job Training (OJT) 


3.8 


5.4 


3.3 


4.4 


2.6 


Job Search Assistance (JSA) 


19.5 


12.4 


17.1 


30.8 


27.3 


Work Experience 


3.2 


1.7 


3.9 


6.5 


5.7 


Miscellaneous Services (a) 


10.0 


9.7 


11.3 


7.9 


1.1 


Sample Size 


4J23 


925 


1,966 


42P 


803 



(continued) 



EXHIBIT 5.3 (continued) 



All Target Adult Adult Male Female 
Service Strategy and Groups Men Women Youths Youths 
Program Service (%) (%) (%) (%) (%) 



OJT/JSA 



Never Enrolled 


43.5 


43.4 


44.6 


41.5 


42.5 


Classroom Training in 
Occupational Skills (CTHDS) 


3.3 


2.1 


5.1 


1.9 


3.3 


Basic Education (BE) 


3.0 


3.6 


2.6 


2.8 


3.1 


On— fli^— TrkK 'Frfttnino ^r^T'F\ 

mc juu 1 nuiui}^ ^ / 


Oft n 


OA A 

zo.o 


Zo.3 




29.9 




Oft 0 




ZO.3 




28.3 






0 J. 


Z.O 


4.Z 


5.2 


\Aic/*ikli An^/\iic QArvrt/^Ac 

jyiiocciiaucoud ocrviccs 


D. / 


ft 


3.0 


O.S 


7.1 










4/Z 


421 


Other Services 












Never Enrolled 




41.1 


37.6 


32.3 


36.9 


Classroom Training in 
Occupational Skills (CT-OS) 


9.4 


4.9 


15.6 


6.5 


9.8 


Basic Education (BE) 


15.7 


6.1 


11.1 


26.4 


29.7 


On-the-Job Training (OJT) 


4.7 


4.7 


5.5 


3.9 


3.9 


Job Search Assistance (JSA) 


19.7 


24.8 


23.4 


12.0 


12.2 


Work Experience 


2.3 


0.9 


2.7 


3.2 


3.4 


Miscellaneous Services (a) 


31.0 


28.4 


31.5 


35.3 


28.5 


Sample Size 


3,064 


1,002 


937 


535 


590 



SOURCE: Adapted from Bloom et al., 1993, which used program enrollment and participation data 
from the 16 SDAs. 

NOTES: Calculations fcr this table are based on data for all treatment group members in the 
IS-month study sample. 

In each service strategy, the total service receipt rate for each target group may be over 
100.0 percent because some treatment group members received more than one service. 

Tests of statistical significance were not performed. 

(a) Miscellaneous program services include such activities as assessment, job-readiness 
preparation, vocational exploration, job shadowing, and tryout employment. 



variation among the target groups. Between 27 and 3 1 percent of each of the four target groups were 
enrolled in on-the-job training (OJT). This small variation among subgroups was somewhat surprising, 
in light of the expected difficulty in arranging on-the-job training for youths, especially male youths. The 
similarity in on-the-job training enrollment rates for youth and adults may be explained by a finding 
reported in Chapter 4: Those youths recommended for the OJT/JSA service strategy appear to have been 
more employable than other youths, as indicated by a much higher control group mean earnings for the 
18-month follow-up period.^ Those youths for whom the SDAs tried to arrange on-the-job training were 
carefully selected to be more job-ready than most. 

Within this subgroup, enrollment in job search assistance was about as common as on-the-job 
training, with approximately 30 percent of the subgroup being enrolled in each service. There was little 
difference among the target groups in this, and for all groups the remaining services were relatively 
uncommon. Approximately 22 percent of this subgroup had been recommended for job search assistance, 
so the observed levels of enrollment in that service were somewhat higher than planned, while the 
enrollment levels for on-the-job training were noticeably lower. In light of the similarity of enrollment 
rates in job search assistance and on-the-job training among the treatment group, this service strategy was 
named OJT/JSA rather than OJT, 

3. Other Services Strategy, As planned, the othei services strategy comprises a mixture of 
services. This reflects the fact that very different types of applicants could be referred to this service 
strategy. It includes the most job-ready applicants, who were recommended only for job search assistance, 
and those for whom pretraining services such as basic education or preemployment skills and exposure 
to the world of woric (included within miscellaneous services in Exhibits 5.1 and 5.3) were appropriate. 
Consequently, the services received varied between adults and youths, with job search assistance, basic 
education, and miscellaneous services all being important. For all target groups, a residual category of 
miscellaneous services was most common, with 3 1 percent of the subgroup enrolled. Job search assistance 
was the second most common service overall, but this was largely because of its heavy use for adults 
(nearly 25 percent were enrolled in this service). For youths, the picture differed: Only 12 percent of 
youths were enrolled in job search, while over 25 percent were enrolled in basic education. 



^Sce Exhibit 4.4. 
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Possible Reasons for These JTPA Enrollment Patterns 



The material just presented suggests that site staff were most successful in foliowing through on 
their initial seivice recommendation in the classroom training service strategy. This is reflected in a higher 
JTPA enrollment rate and a preponderance of classroom training in occupational skills among those who 
did enroll. In the other strategies, the JTPA enrollment rate for the treatment group was lower and — in 
the case of the OJT/JSA category - the focus on the intended "anchor" service (on-the-job training) was 
less than expected. 

These patterns of enrollment to some extent are related to the challenges local staff faced in 
serving applicants. While making a match between a client and a classroom training provider is not 
always easy, the JTPA agency and the service provider have similar missions (providing training) and may 
well see each other as continuing partners. JTPA may be a major funder of training in a community, 
especially at community-based organizations and other agencies without major state government grants-in- 
aid. JTPA staff report that they often know the details of entrance requirements for different types of 
training and use this in developing recommendations for individual clients. In some jurisdictions, training 
agencies may even accept the test results and assessment made by JTPA staff *° These factors suggest 
that a relatively high proportion of the classroom training program group will be placed with a training 
agency and enrolled in JTPA. 

The process for developing an OJT position differs substantially and in ways likely to lead to 
lower rates of enrollment - both in OJT and in JTPA overall - for the OJT/JSA service strategy. Private 
employers* main goal is producing goods or services, not subsidized training. They must be induced to 
participate as employers and trainers through a combination of financial incentives and assurances that the 
administrative costs will not overwhelm the subsidy they receive. Even in small programs, local staff have 
to develop OJT positions with many different employers. With minority clients, staff may face employer 
discrimination, which can be subtle and hard to document, and employers considering offering an OJT 
position typically retain the hiring authority. Finally, clients are likely to be actively seeking employment 
(as those slated for the classroom training strategy often are not), either on their own or through other 
program services such as job search assistance. If a client does find an unsubsidized job, it could result 



'^Service providers may be hesitant to do ihis when they are paid through a performance-based contract with a 
substantial proportion of iunds tied to successful completion of training and placement in a job. 
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in no JTPA enrollment (if the placement was unrelated to the program) or to an enrollment in job search 
and a placement. 

These operational realities also affect the experiences of those members of the treatment group 
who were never enrolled in JTPA, as reflected ixi the small survey of nonenrolled treatment group 
members discussed in Chapter 4." In the classroom training strategy, most of those who were never 
enrolled in JTPA either lost contact with the program (28 percent), were no longer interested in JTPA (19 
percent), or did not receive any substantial services from local staff (28 percent). In this service strategy, 
for the remaining 23 percent of the nonenrolled, local staff woriced unsuccessfully to arrange a service 
(usually classroom-based) and enroll the client. 

The situation for OJT/JSA is in sharp contrast. Staff woriced unsuccessfully with 68 percent of 
the nonenrolled to arrange a service and enroll the client.*^ Fifty-seven percent of this sample of the 
nonenrolled were referred to employers for a potential on-the-job training position, while 23 percent 
received other help finding a job. 

The situation for the other services subgroup is a mixture of the other two (as expected, given the 
variety of people recommended for this service strategy), and the story behind the results is less clear. 
Local staff worked unsuccessful!} with 49 percent of the nonenrolled to arrange a service and enroll the 
client. Much of this activity concerned efforts to place more job-ready clients in a job: Twenty-five 
percent of the nonenrolled were referred to employers for possible on-the-job training, while 27 percent 
participated in some form of job search assistance. The next most common activity was further 
assessment of needs and interests and counseling, as would be likely for less job-ready clients, for whom 
it might be unclear what the best service plan would be. 

This provides supporting evidence that arranging placement in a service that resulted in enrollment 
in JTPA was easiest in the classroom training service strategy. The JTPA enrollment rate is highest for 
those recommended for this strategy, and the rate at which staff work with clients without producing an 
enrollment is lowest (23 percent rate for classroom training versus 68 percent for OJT/JSA and 49 percent 
for other services). 

To summarize the situation for the treatment group: Nonenrolled post-random assignment 
involvement in JTPA was most common among those in the OJT/JSA service strategy and least common 

* 'Other findings from this survey were reported in Exhibit 5.5. For more detail on this issue, see Kemple, 
Doolittle, and Wallace, 1993, Chapter 3. 

'^Of the remaining 32 percent, staff lost contact with 5 percent, 10 percent were no Ic.iger interested, 16 percent 
did not receive any substantial services from local staff, and one percent turned out not to be eligible. 



among those in the classroom training strategy. Since JTPA enrollment rates follow an opposite 
pattern,'^ when service to nonenrolled treatment group members is combined with service to those 
enrolled, differences in involvement with JTPA among the service strategies diminish. Those treatment 
group members in the OJT/JSA strategy were slightly more likely (86 percent) to have had some post- 
random assignment involvement with JTPA than were people in the other strategies (81 percent for other 
services and 79 percent for classroom training). 

E. The Implications of These Patterns of JTPA Enrollment for Impact Analysis 

Two key points emerge from this discussion of JTPA service receipt. First, because a substantial 
proportion of the treatment group never enrolled in JTPA, it is useful to present two types of impact 
estimates. Second, while the service strategies ended up with a slightly different emphasis than planned, 
they are distinct service approaches. 

1. Alternate Impact Estimates , The decisions on the point of random assignment and the 
resulting pattern of JTPA enrollment increased the importance of providing alternative impact estimates: 

• A pure "experimental" comparison of the treatment group and the control 
group, which measures the impact of providing access to JTPA services to those 
people who local staff recruited, assessed, and found eligible and appropriate 
for the program. 

• A conversion of this estimate to an approximate measure of the impact of 
JTPA on people who actually received services, i.e., adjusting for the fact that 
some treatment group members never received program services. 

Calculating the pure experimental impact estimate is relatively straightforward: This estimate is 
the difference in the average outcomes for treatment group members compared with the averages for 
control group members. These impacts are labeled "impacts per assignee" in the accompanying impact 
report and stand for "impacts per person randomly assigned to the 18-month treatment group sample."*^ 
This treatment-coiitrol group comparison rests solely on the validity of random assignment, which assures 
that there were no systematic differences between the two groups at random assignment except that the 
treatment group was then given access to JTPA services and the control group was not. 



"Enrollment rates were highest in classroom training and lowest in OJT/JSA, as shown in Exhibit 5.2. 
'^The sample for this special study was too small to allow simultaneous disaggregation by target group and 
service strategy. 

"Bloom et al., 1993. 
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Ideally, one would convert this measure of the impact of the offer of access to JTPA to an 
estimate of the impact of receiving JTPA services, i.e., "impacts per service recipient." However, two 
types of is;>ues arise in this effort. 

On the data side, the only measure of service receipt in JTPA available for the entire sample 
comes from the local SDA enrollment records. The results reported earlier on the involvement of 
nonenrolled treatment members with JTPA imply that enrollment records understate the receipt of JTPA 
assistance. The small sample size of this special study of nonenrolled treatment group members makes 
it impossible to use these data to develop a more accurate measure of service receipt for the full sample; 
thus, enrollment has to be used as a measure of "participation" in the program. 

Putting aside these data problems, there is no technique for directly calculating impacts per 
enrollee that is as reliable as the pure experimental calculation of impacts per assignee. Enrollees in JTPA 
are not a random sample of all treatment group members, so it is incorrect to compare the experiences of 
enrollees with those of the entire control group. At this time, there is no reliable way to identify the 
control group counterparts of people in the treatment group who ended up enrolling in JTPA, so a 
different approach must be tried. 

The precise details of the underlying assumptions and the calculations used to convert impacts per 
assignee to impacts per enrollee are discussed in the accompanying impact report, but the approach used 
assumes that all program impacts accrue to those who participated in JTPA (i.e., that nonparticipants were 
unaffected by JTPA) and that enrollment is a good measure of participation. Impacts per enrollee are, 
therefore, "scaled up" impacts per assignee, reflecting the assumption that enrollees are the source of all 
program impacts. As an illustration of this conversion from impacts per assignee to impacts per enrollee, 
if the program affects only enrollees, and if half of the treatment group enrolls in the program, then 
impacts per enrollee will be twice those per assignee.'^ 

This implied impact per enrollee rests on an untestable assumption that nonenrollees are unaffected 
by the program. However, the finding of nonenrolled service receipt illustrates the need to question this 
assumption. 



'^Such attempts encounter the types of analytical difficulties that earlier nonexperimental impact studies 
experienced, which led the Department to decide on a random assignment impact study. The nonexperimental 
component of this project is attempting to develop new methods to address these problems. 

*^The conversion is made by dividing impacts per assignee by the enrollment rate. In this example, division by 
.5 doubles impacts. See Bloom et al., 1993, for the details of this procedure. 
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Recognizing this need for caution, the accompanying impact report presents impacts per assignee 
and per enrohee, but argues that these "bracket" the ideal estimate: impacts per service recipient. The 
only assumption needed for this bracketing argument to hold is that any program impacts on nonenrolled 
"service recipients" do not have a different sign than the impacts on enrollees. For example, if program 
impacts on enrollees are positive, the impacts on nonenrollees must be zero or positive. If this assumption 
holds, the "intensity" of the services received by nonenrollees and the size of the resulting impacts 
determine the "width" of the bracket: Typically, the less intensive the services, the smaller the bracket. 
The findings on nonenrolled involvement in JTPA, reported earlier, are on the whole consistent with this 
view, since most of the service for the half of the nonenrollee semple receiving some JTPA assistance is 
clearly not intensive. However, in instances where individuals are referred to job club, they could have 
participated in the same services that led to enrollment for those who found a job. 

2* Service E mphases Among the Strategy Subgroups , While the enrollment patterns of the 
treatment group led to some revisions in the descriptions of the three service strategies, distinct service 
emphases did emerge for each. Exhibit 5.4 summarizes the JTPA enrollment data for the treatment group 
members by presenting information on the two key services in each service strategy, by target tjoup. The 
top panel presents data for all members of the treatment group; e.g., 61 percent of all adult male treatment 
group members recommended for the classroom training service strategy were enrolled in CT-OS, basic 
education (BE), or both. The bottom panel presents data just for those who enrolled in JTPA, i.e., the 
percentage of enrollees receiving the two key services. 

Both measures show a clear emphasi.^ for each service strategy. The lower panel on enrollees 
shows the following emphases: 

• Classroom training. Most enrollees were active in classroom-based services 
of some type, and, as discussed earlier, classroom training in occupational skills 
was much more important than basic education. Eighty-six percent of adult 
male enrollees, 89 percent of adult female enrollees, 80 percent of male youth 
enrollees, and 86 percent of female youth enrollees were active in CT-OS, basic 
education, or both. 



^*To illustrate why this is true, assume again that 50 percent of the treatment group were enrolled, so that a 
conversion from impacts per assignee to impacts per cnrollce would double the size of the impacts. However, if an 
additional 25 percent of the treatment group, c.g., received some JTPA service without enrollment, then 75 percent 
of the treatment group were actually service recipients (50 percent enrolled, 25 percent not). Impacts per service 
recipient would be only 50 percent larger than impacts per assignee. 
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EXHIBIT 5.4 



KEY SERVICES FOR TREATMEhfT GROUP MEMBERS AND ENROLLEES, 
BY TARGET GROUP AND SERVICE STRATEGY 



Sample and 
Target Group 



Classroom 
Training 



OJT/JSA 



Other 
Services 



Treatment 

GroTO Members 



Adult Men 



61 % enrolled in 
CT-OS or BE 



49% enrolled in 
OJT or JSA 



52% enrolled in 

JSA or misc. services (a) 



Adult Women 



65% enrolled m 
CT-OS or BE 



49% enrolled in 
OJT or JSA 



52% enrolled in 

JSA or misc. services (a) 



Male Youths 



60% enrolled in 
CT-OS or BE 



49% enrolled in 
OJT or JSA 



56% enrolled in 

BE or misc. services (a) 



Female Youths 



62% enrolled in 
CT-OS or BE 



49% enrolled in 
OJT or JSA 



50% enrolled in 

BE or misc. services (a) 



Enrollees 
Adult Men 

Adult Women 

Male Youths 

Female Youths 



86% enrolled in 
CT-OS or BE 

89% enrolled in 
CT-OS or BE 

80% enrolled in 
CT-OS or BE 

86% enrolled in 
CT-OS or BE 



87 % enrolled in 
OJT or JSA 

88% enrolled in 
OJT or JSA 

85% enrolled in 
OJT or JSA 

85% enrolled in 
OJT or JSA 



89% enrolled in 

JSA or misc. services (a) 

82% enrolled in 

JSA or misc. services (a) 

83 % enrolled in 

BE or misc. services (a) 

80% enrolled in 

BE or misc. services (a) 



SOURCE: MDRC calculations from Background Information Form responses and from 
program enrollment and participation data from the 16 SDAs. 

NOTES: Calculations for this table are based on data for all treatment group members and 
enrollees in the 18-month study sample. 

(a) Miscellaneous program services include such activities as assessment, job-readiness 
preparation, vocational exploration, job shadowing, and tryout employment. In this table, 
miscellaneous program services also include work experience, which is identified as 
a separate program service in other reports on the National JTPA Study. 

I 'Jo 



• OJT/JSA. Most enrollees were active in OJT or job search assistance. Eighty- 
seven percent of adult male enrollees, 88 percent of adult female enrollees, 85 
percent of male youth enrollees, and 85 percent of female youth enrollees were 
active in OJT, job search assistance (JSA), or both. 

• Other services. The predominant types of services differed for adults and 
youths. Eighty-nine percent of adult male enrollees and 82 percent of adult 
female enrollees were active in job search assistance or miscellaneous services, 
while 83 percent of male youth enrollees and 80 percent of female youth 
enrollees were active in basic education or miscellaneous services. 



n. Measuring Non-JTPA Service Receipt and the Treatment-Control Service Difference 

The follow-up survey conducted for members of both the treatment and control groups was the 
sole source of information on receipt of non-JTPA services. It asked respondents about spells of 
education, training, and other employment-related services they had received lasting longer than one week. 
This minimum service duration was chosen because of likely respondent recall problems for very short 
services. It also asked about assistance respondents had received in searching for woric during their last 
spell of unemployment.^^ Since respondents were often unable to identify the funding source for training 
(and hence could not distinguish between JTPA and other support), it was not possible to separate JTPA- 
fiinded and other services. Finally, the survey did not include questions on several of the JTPA activities 
grouped into the category "miscellaneous services," which proved to be important within the other services 
strategy because they were rarely provided outside JTPA. This prevented an estimation of the treatment- 
control service difference for this category of service. 

To summarize ttie effort to collect service receipt data: It was possible to collect reasonably 
complete survey data on the receipt of classroom training in occupational skills and basic education for 
both the treatment and control groups. SDA data on the receipt of on-the-job training and paid work 
experience through JTPA were also available, but respondents to the survey could not or did not 
distinguish these activities fix)m regular employment. However, JTPA participation in these activities by 



^'This section is based on Chapters 4-7 of Bloom ct al., 1993, which presents the details of this service receipt 
analysis. 

^^e survey did not attempt to collect information on job search efforts during alt post-random assignment 
periods of unemployment because of serious concerns about the ability of respondents to recall the details of prior 
job searches. Even with the question limited to the most recent spell of unemployment, there was evidence that 
certain types of job search assistance (most importantly, use of the state Employment Service) were systematically 
underreported. Without a complete post-random assignment history of job search assistance from the survey, it was 
not possible to calculate a job search assistance service receipt rate for the control group. 
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the treatment and control groups is probably a reasonable indicator of the treatment-control service 
difiference because JTPA is the primary source of these activities. Finally, there was no reliable source 
of information on non-JTPA sources of job search assistance and miscellaneous services, so it is not 
possible to report a service difference for these activities. However, as mentioned in Chapter 2, members 
of the control group in several sites probably were served by state Job Service staff who were co-located 
with JTPA. 

Exhibits 5.5 through 5.8 summarize two key indicators of the service difference between the 
treatment and control groups: the percent of the treatment and control group receiving each type of service 
and the average number of hours each group participated in specific services.^^ The entries in these 
tables for the hours of activity are the average hours of participation for the treatment and control groups 
(including zeros for those who did not participate in a service). Since the averages for the two groups 
are **diluted** by the high proportion of each group that did not participate in the individual services, the 
differences appear smaller than might be expected. Thus, the entries represent two alternative measures 
of the difference in average participation for the treatment and control groups, which corresponds to the 
service difference for which impacts were measured. 

Clear patterns of service differences do appear in the data: 

• For the classroom training service strategy, clear treatment-control group 
differences in the key service (classroom training in occupational skills) did 
emerge for all target groups. Female youths had the largest difference in this 
service, primarily because their treatment group members had much higher 
hours of participation than did the other target groups, while the hours for the 
respective control groups were fairly similar. For adults and male youths, the 
treatment and control group average hours of basic education were very similar; 
female youths in the treatment group averaged 36 more hours in this service 
than did their control group counterparts, again a doubling of the participation 
in this service. 

• For the OJT/JSA service strategy, it was estimated that the treatment group 
was much more likely to participate in OJT than the control group and averaged 



^^The calculations behind these data are explained in Bloom et al., 1993, Chapters 4-7. As discussed in the text, 
the calculations rely on data from two sources: the 1 8-month follow-up survey for classroom activities and the SDAs' 
enrollment and termination records for OJT and work experieiice. The survey included questions on hours of 
participation, while SDA records do not. The research team assumed that OJT and work experience participants 
worked a full-time job of 40 hours per week and that there was no non-JTPA source for thes^ services. As 
discussed in the text, there was also no reliable source of information on participation in non-JTPA job search 
assistance and miscellaneous services, so a comparison of hours was not possible. 
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substantially more hours of OJT than did the control group for each target 
group.^^ Here, no target group had a markedly different estimate. Because 
of the problems of measuring non-JTPA participation in job search assistance 
(discussed above), it was not possible to calculate a service receipt or hours 
difference for this activity. 

• For the other services strategy, the problems encountered in measuring non- 
JTPA program participation in job search assistance and miscellaneous services 
(which are the most common services for adults) prevented calculation of 
service differences for these two groups. For youths, basic education replaces 
job search assistance as an important service.^^ Interestingly, for female 
youths there was virtually no difference in hours in basic education between the 
treatment group and controls, while for male youths the control group actually 
participated more. This was probably because of the numerous non-JTPA 
sources of basic education in most communities and JTPA*s emphasis; on 
services that are more directly related to employment. , 

This chapter's discussion of services received by the treatment and control groups reinforces three 
central points. First, most of the treatment group got JTPA services. Slightly over 60 percent of the 
treatment group were enrolled in JTPA services following random assignment, one-sixth probably received 
some type of service from JTPA without enrollment in the program, and the remaining one-fifth received 
no assistance. Second, the service strategies analyzed in the accompanying impact report did have distinct 
service emphases for all target groups. Classroom training had the intended focus on classroom-based 
services, OJT/JSA ended up emphasizing immediate placement in some type of job, and other services 
was a mixture of miscellaneous services or job search assistance for adults and miscellaneous services or 
basic education for youths. Finally, control group members did receive services outside of JTPA, so the 
impact estimates are for the incremental services received by the treatment group above the level received 
by the control group. The service differences between the treatment and control groups are clear — 
though modest in size — for classroom training in occupational skills and OJT. For job search assistance 
and miscellaneous services it was not possible to estimate service differences. 



^^As stated earlier, this conclusion rests on the assumption that JTPA is the primary source of funding for on-the- 
job training and work experience. 

^^Youths in general had much higher hours of participation in basic education than did adults; e.g., male youths 
in the control group averaged 79 hours in basic education compared to only 10 for adult males and 26 for adult 
females. 
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CHAPTER 6 

SAMPLE SIZE. DATA COLLECTION. AND RELATED ISSUES 



The National JTPA Study relied on a variety of data sources in describing the study sites and the 
services received by sample members and in estimating program impacts. This chapter discusses two key 
data-related issues encountered in the study: (1) the size of the sample used for the analysis of program 
impacts, and the resuhing ability of the research team to estimate impacts for target groups and service 
strategies, and (2) the extent of survey response bias for the four target groups and the method used to 
adjust impact estimates where such problems existed.* jr 




Both issues are related to the internal validity of the impact estimates. If the sample is too small 
for detecting policy-relevant impacts (if they occur), the impact analysis might incorrectly conclude that 
the program made no difference in the study sites when in fact it did. Further, if a bias was introduced 
into the analysis because of the way in which data were collected, that could lead to incorrect conclusions 
about program impacts in the study sites. 

The data sources for the study, reviewed in more detail in Chapter 2, include: (1) the Background 
Information Form (BIF), completed by applicants and local staff prior to random assignment; (2) the 
Random Assignment Record (RAR), created as part of the random assignment phone call; (3) JTPA 
enrollment, participation, and termination data from existing state or local management information 
systems; (4) administrative records on JTPA programs, including client and service characteristics and 
program costs; (5) field research on the study SDAs, during site selection and implementation of the study; 

(6) data from the Job Training Quarteriy Survey (JTQS) on the characteristics of JTPA clients nationally; 

(7) the first (18-month) follow-up survey, which collected information on post-random assignment 
outcome data; and (8) earnings data from state Unemployment Insurance (UI) agencies, providing 
quarterly data on wages and salaries for most workers in the sample.^ 



'This miiterial is discussed in more detail in Appendices to Bloom et al., 1993, and Kemple, Doolittle, and 
Wallace, 1993. 

^As noted in Chapter 2, usable UI wage data had not been obtained from Ohio or New Jersey at the time the 
analysis was done, and data from Montana were incomplete. In addition, some types of workers are not included 
in the Unemployment Insurance system. 




6-1 




I. The Sample for the 18-Month Impact Analysis 

The overall experimental sample of 20,601 is large, as is the number of completions of the 18- 
month follow-up survey (14,441). As discussed in Chapter 2, the difference between the overall 
experimental sample and the 1 8-month follow-up survey sample arises because of the exclusion from the 
interim analysis of 473 "extra" treatment group members, 3,097 people whose follow-up interview was 
scheduled for fewer than 18 months after random assignment, and 2,584 people who did not respond to 
the 18-month survey. 

A sample of more than 14,000 persons is clearly adequate for estimating the impacts of the overall 
JTPA program for the full sample. The real question is how far this sample can be disaggregated into 
the four target groups, the service strategies, and further subgroups. Not surprisingly, despite this large 
sample, it is not possible to answer all possible impact questions for all subgroups. This section describes 
the concept used in the study to assess the ability of the researchers to answer key impact questions with 
the available sample. It then presents information on the central outcome of the 18-month impact report: 
cumulative earnings over the entire 18-month follow-up period. 

The concept of "minimum detectable effects" (MDEs) was used to assess the ability of the 
available sample to answer key impact questions. MDE is defined as the minimum impact on an outcome 
that can be detected (i.e., distinguished from zero or no impact) with the available sample and the desired 
level of confidence in the estimate.^ It can be expressed as an absolute amount (e.g., an impact of 
$1,000) or as a percentage difference (i.e., earnings 10 percent higher than the control mean). The MDE 
typically decreases as the sample increases (e.g., with a sample of 10,000, researchers would be able to 
detect smaller impacts than with a sample of 5,000). It also decreases as the variance in the outcome 
within the sample decreases. 

The MDE concept can be used to assess whether it is possible to detect "policy-relevant" impacts 
(i.e., impacts large enough to "matter" in the policy process) should they occur. There are many ways 
to decide on the policy relevance of impacts, so readers may assess the size of the MDEs discussed below 
in different ways.^ If the MDE is larger than a policy-relevant impact (e.g., if the MDE is a $1,000 
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^The test of statistical confidence used for most of the impact estimates presented in the accompanying report 
(Bloom et al., 1993) is 90 percent confidence using a two-tailed test with 80 percent power. See Bloom et al., 1990, 
for the details of how this concept is calculated. 

^See Abt Associates Inc. et al., 1989, for a discussion of the various approaches. 
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impact and a $500 impact would "matter"), then the analysis would incorrectly imply that the program 
did not make a difference when in fact it did make enough difference to matter. 

With this as background, it is possible to assess how much the total sample can be disaggregated 
and still provide policy-relevant conclusions. Exhibit 6.1 presents minimum detectable effects (MDEs), 
using the 18-month survey sample, for 18-month earnings impacts per person assigned to the treatment 
group (labeled "per assignee) for the various combinations of target groups and service strategies in the 
study. This is the central outcome examined in the accompanying 18-month impact report.^ 

The minimum detectable effects on earnings for JTPA overall for aduh men and aduk women are 
$855 (6.8 percent) and $460 (5.9 percent), respectively.^ This means that impacts smaller than these 
amounts could not be detected (i.e., could not be distinguished from zero or no impact) with the required 
level of confidence. For the female and male youths samples, each of which is about half the size of the 
two adult target groups, the minimum detectable effects are $655 (10.3 percent) and $1,075 (9.5 percent), 
respectively. For adults, the minimum detectable effects for the service strategy subgroups are in the 9 
to 15 percent range, while for youths, they are in the 15 to 20 percent range. 

Not surprisingly, despite the large size of the overall sample, there are clear limits to the precision 
of the impact estimates presented for the many possible combinations of target groups and service 
strategies. Although the MDE for JTPA overall for each target group is 10 percent or less, once the 
sample is disaggregated by both target group and a subgroup (such as by service strategy), the MDE rises 
noticeably. 

n. Survey Response Bias 

While the response rate to the survey was high (85 percent overall and between 80 and 88 percent 
for the four target groups), survey nonresponse bias could still arise if outcomes for respondents differed 
systematically from outcomes for those who did not complete the survey.^ Because the study has multiple 



*The MDE for this outcome, in percentage terms, will be larger than for "yes/no" outcomes (such as the 
percentage of the treatment and control groups ever employed during the period) because they have a smaller 
variance. However, the MDE for 18-month cumulative earnings is smaller than for individual monthly earnings 
impacts. 

^otc that the adult males have a much higher control group average and much greater variance in control group 
earnings. 

'What matters is not just whether there are differences in outcomes between respondents and nonrcspondcnts, 
but whether outcomes differ (in ways that cannot be corrected for) between respondents and nonrcspondcnts within 
the treatment group and within the control group. 
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EXHIBIT 6.1 



ACTUAL MINIMUM DETECTABLE EFFECTS PER ASSIGNEE 
FOR ESTIMATES OF PROGRAM IMPACTS 
ON TOTAL EARNINGS DURING THE 18-MONTH FOLLOW-UP PERIOD 
(FOR A TWO-TAILED TEST AT THE 0.10 LEVEL WITH 80 PERCENT POWER) 

Target Group In Nominal As a Percentage of the 
and Service Strategy Dollars Control Group Mean 

Adult Men 

Classroom Training 1,765 14.8 

OJT/JSA 1,178 9.1 

Other Services 1,795 14.2 

All Strategies 855 6.8 

Adult Women 

Classroom Training 653 9.8 

OJT/JSA 793 8.9 

Other Services 1,098 13.5 

All Strategies 460 5.9 

Male Youths 

Classroom Training 1 ,948 20.0 

OJT/JSA 1,990 14.1 

Other Services 1,733 17.3 

All Strategies 1,075 9.5 

Female Youths 

Classroom Training 893 14.7 

OJT/JSA 1,560 20.1 

Other Services 1,185 20.8 

All Strategies 655 10.3 



SOURCE: National JTPA Study impact team calculations. 
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data sources for earnings, it is possible to analyze whether the exclusion of survey nonrespondents appears 
likely to have affected study findings.* 

The study team tested for survey nonresponse bias in the impact estimates for earnings using 
Unemployment Insurance earnings data for both survey respondents and nonrespondents in 13 of the 16 
study sites.' The team constructed two sets of impact estimates using UI earnings data for the first four 
quarters following random assignment: one for those who had responded to the 18-month survey and one 
for both respondents and nonrespondents. The comparison allowed an assessment of whether the 
difference in results was so large as to require further statistical adjustments to address response bias. The 
research team judged the estimated bias for adult men (-$26) and youths (-$38) to be so small as to be 
acceptable, but decided that the estimated bias for adult women ($86) needed to be addressed.'" 

The approach adopted to address the response bias for adult women again took advantage of the 
two available data sources on earnings. It involved using UI earnings data to impute values for survey 
nonrespondents (or for respondents who provided insufficient information to determine employment status 
in all months)." Imputations were developed for the adult female target group in estimating treatment 
and control means and impacts on earnings, employment rates, weeks and hours worked, and other out- 
comes. These imputations made little difference in the control group mean for 18-month earnings (-$19), 
the central outcome analyzed in the accompanying 18-month impact report, but they lowered the estimated 
impact by $106, or 16 percent. They also lowered the estimated impact on employment rates, weeks 
worked, and hours worked by 15 to 20 percent. 



'This section is based on material in Appendix D and Appendix E in Bloom et al., 1993. 
'Data were not available for Butte, Jersey City, and Marion. 

">The nonresponse bias for adult women may be concentrated among the treatment group. The UI data showed 
quite similar mean earnings for control group responders and nonresponders, but a $1,204 difference for treatment 
group members. However, it is also possible that the response bias is creating a difference in treatment group mean 
earnings between respondent and nonrespondent treatment group members and masking a difference for control 
respondents and nonrespondents. 

"This was possible in only 14 sites, since no UI data were available for Marion or Jersey City. Data from 
Montana were incomplete but appeared to be accurate, so they were used in the imputations. See Section 10 of 
Appendix D in Bloom et al, 1993, for the procedures used to make the adjustments. 
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CHAPTER 7 

SITE SELECTION AND THE EXTERNAL VALroiTY OF THE STUDY 



The goal of this chapter is to help the reader assess the extent to which the findings of the study 
can be generalized to the JTPA system as a whole. It begins by briefly discussing the process by which 
sites were selected and then compares key features of the local setting and programs of the service 
delivery areas (SDAs) in the study with national averages for all SDAs.' The chapter concludes by 
comparing the characteristics of the sample analyzed in this and the accompanying 18-month impact 
report^ to the characteristics of JTPA participants nationally. This will assist the reader in assessing the 
•'external validity" of the study findings, i.e., the extent to which they apply to the larger JTPA system. 

The chapter's conclusions are relatively straightforward. The sites are not a random sample of 
all SDAs, but they resemble the national JTPA system in many ways and include much of its diversity. 
Although they were volunteers, sites did not know anything about the impacts of their programs prior to 
the study (since the information on a control or comparison group was not available). The factors that 
appeared to lead them to participate do not appear to be related to program impacts. Importantly, the 
success of the study sites in meeting or exceeding the U.S. Department of Labor^s JTPA performance 
standards (a performance measure that some might have expected to be related to impacts) was very 
similar to all programs nationally. And, in fact, site success in meeting the Department's performance 
standards turned out to be unrelated to actual site impacts among the 16 study sites. Despite the observed 
similarities between the study sites and the national JTPA system, they nevertheless do not statistically 
represent all SDAs nationally. 

L The Meaning of External Validity 

In a strict statistical sense, study findings are externally valid when a sufficient number of sites 
are randomly chosen and, hence, the findings are representative of what would have been found had all 
programs been studied. Prior to this study, to the best of the knowledge of the researchers working on 



*For a more detailed discussion of these issues, see Doolittle and Traeger, 1990; Kcmple, Doolittle, and Wallace, 
1993; Bloom, 1991; and Bloom et al., 1993. 
^Bloom et al., 1993. 
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the project, no comparable social experiment had used this approach 



1.^ The demands on sites of 



participating in a social experiment are substantial enough, as discussed earlier in this report, that no one 
had even attempted this, recognizing that many possible sites would not want to or be capable of 
participating/ 

More commonly, experiments had been done in one or a small number of sites that volunteered 
to participate.^ In studies testing the potential impact of innovations, researchers have sought out strong 
sites with a capacity to implement the program model under review, to assure that the study provided a 
"fair test" of the suggested approach.^ However, in studies of the effectiveness of ongoing programs. 



researchers do ilbt wish to systematically select sites that are likely to produce unusually strong or weak 



In this situation, in selecting sites, researchers keep in mind several criteria, which together serve 
as practical guidelines for the extent to which they have achieved "external validity":' 

« Process of selection. Was the process used to recruit and select sites designed 
or implemented in a way likely to recruit sites that would produce unusually 
strong or weak findings?* 

• Geographic and "environmentar* diversity. Do the sites selected reflect the 



^Aftcr the beginning of this study, Abt Associates Inc. conducted a study of the recently enacted, mandatory 
employment and training program for food stamp recipients subject to a participation requirement. In this study, the 
Abt research team selected a random sample of 60 local food stamp programs for recruitment into the study; 47 
agreed to participate. An additional sfac sites were selected and the study was done in 53 sites. The differences in 
the features of JTPA and the food stamp employment and training programs were substantial. JTPA was a voluntary 
program with recruitment problems, while the food stamp program was mandatory; JTPA was an ongoing program, 
and the food stamp program was new; JTPA offered individualized services, while the food stamp program was 
largely job-search-based; and many more interests were represented in the governance structure of JTPA at the local 
level than was the case for the food stamp program. See the discussion in Chapter 2 of this report and Puma et al., 
1990. 

*Even in medical experiments, in which random assignment procedures have long been used, subjects are 
typically not selected to be representative of all potential persons who would be served by the procedure under study. 

*The Negative Income Tax (NIT) experiments, e.g., operated in fewer than 10 sites. 

^Examples of this include the Supported Work and JOBSTART demonstrations organized by MDRC, the Summer 
Training and Employment Program (STEP) Demonstration organized by Public/Private Ventures, and the AFDC 
Homemaker-Home Health Aide Demonstrations studied by Abt Associates Inc. 

^The fectors listed below are similar to those cited by many other researchers. The actual list is based on 
comments by Rebecca Maynard at an April 1990 conference on program evaluation sponsored by the U.S. 
Department of Health and Human Services. Dr. Maynard was a discussant for a p^r on the National JTPA Study 
that dealt with the criteria for external validity (Hotz, 1992). Many of the papers from the conference, including 
Hotz's, were published in Manski and Garfinkel, 1992. 

•Obviously, random selection of sites does not fall into this trap. That is its major advantage. However, with 
relatively small numbers of study sites and many possible sites in the universe from which they are selected, a 
random selection of sites could do pooriy on the other criteria. 




impacts. 
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geographic diversity of the overall ''universe" of sites? Do they include the 
diversity of labor market conditions (unemployment, employment base, wages) 
seen nationally? Do they include a mix of urban, suburban, and rural locales 
and both larger and smaller population centers? 

• Operational diversity. Do the sites selected reflect the diversity in 
administrative structure, program emphasis, client characteristics, and richness 
of funding and services seen in the national program? 

• Operational maturity and stability. Are the sites established programs, rather 
than new entities and recently reformed programs, and are they under any 
unusual administrative stresses that would make them atypical JTPA programs? 

The remainder of this chapter describes the process of site selection in the National JTPA Study, and 
compares characteristics of the resulting sites and research sample to their national counterparts. 

n. The Site Selection Process 

The Advisory Panel for the evaluation of JTPA recommended that the Department conduct a series 
of classical experimental studies of the impacts of JTPA for selected target groups and activities in SDAs 
that volunteered to participate.^ 

Wanting to increase the generalizability of study findings, the Department specified in its Request 
for Proposals that contractors should seek a representative sample of SDAs by selecting sites to recruit 
through a probabilistic sampling process.*^ However, despite its increased emphasis on generalizability 
of the findings, the Department did not wish to mandate SDA participation in the study. 

Two additional constraints on site selection were also important. First, because of budgetary 
limitations, no more than 20 SDAs could be included in the study. Second, the selected SDAs, as a 
group, had to provide a large enough research sample to permit analysis of many types of program 
impacts for the full sample and important subgroups, as noted in Chapter Ts discussion of the goals of 
the study. This meant that very small SDAs were not selected." Finally, because of the complexity of 



*Job Training Longitudinal Survey Research Advisory Panel, 1985, pp. 21-26. 

^^asically, the Department wanted all SDAs to be divided into groups by factors such as size, region, and 
performance on the existing program standards. Then, within these groups, selection was to be done on a 
probabilistic basis. 

"The team thus excluded SDAs with fewer than 500 terminees in JTPA program year 1984, the most recent year 
for which data were available at the start of the study. Although these SDAs amounted to about one-fifth of all local 
programs, they served about 7 percent of all JTPA participants. 
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the research design, SDAs had to have the administrative edacity to follow the study's procedures.'^ 

The sites in the study were recruited to participate in late 1986 and 1987, in three phases.'^ 
Under the initial probabilistic site selection plan, and with the original research design,*^ the researchers 
contacted 83 sites about participating in the study; 5 of these eventually agreed to participate.'^ Given 
this initial low acceptance rate, the Department and the researchers agreed that a probabilistic sample of 
sites was not possible. The Department then allowed the researchers to recruit any site within groupings 
defined by SDA size and region, but the research design remained unchanged. During this second phase, 
the researchers contacted an additional 61 sites, 4 of which eventually agreed to participate.'* 

Following this experience, the Department and the researchers made four key revisions to the 
research design, the first three of which reduced the burden (including the cost) of participation for sites 
and the fourth of which increased the benefits: 

• Shorten from 30 to 18 months the period during which members of the control 
group could not be served by study sites, to lessen the public relations 
problems connected with denying access to services. 

• Redefine the service strategies for which impacts were to be estimated to more 
closely reflect usual SDA assessment and service provision practices.'^ 

• Change the random assignment ratio firom one treatment group member per 
control group member to two per control group member to lessen the intake 
burden on sites and help them meet recruiting targets.'* 



'^The Department and the research team eliminated some SDAs on this ground because they had serious legal 
or administrative problems, which would have jeopardized implementation of the study. In addition, many of the 
sites that declined to participate did so, in part, because of the difficulty of implementing the research design in their 
program. 

''For a more detailed discussion of the entire site selection process and the changes made in the research design 
as site selection progressed see Doolittle and Traeger, 1990. 

'^Revisions of the rese.'^:h design over time are discussed below. 
*^Four of these acceptances came after the research design was revised. 
**One of these ac'x.'ptances came after the research design was revised. 
^^The defmitions of the fmal service strategies are discussed in Chapter S. 

^*This change lessened the burden on sites because it reduced the number of people they had to recruit, determine 
eligibility for, and assess to meet their enrollment targets. The shift in the random assignment ratio from the 
statistically optimal one-to-one ratio resulted in an ^proximately 12 percent decline in the statistical precision of 
the overall impact estimates. In light of this small decline in precision and the substantial benefit to sites, the 
Department approved the change. However, a shift to a three-to-one ratio for the entire sample was rejected by the 
Department and the research team because it would have reduced statistical precision by 33 percent. 
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• Increase the compensation paid to study sites from an average of $40,000 to 
$170,000 to defray more of the costs of participating in the study. 

The third phase of recruitment followe ' the making of these changes. The researchers were able 
to recruit additional sites, although, even with the e changes in the design, the "participation rate" among 
SDAs contacted did not exceed 10 percent. During this final phase, virtually all sites willing and able to 
participate were accepted into the study. The 16 study sites began random assignment between November 
1987 and October 1988, and all sites completed random assignment by the end of September 1989. 

Exhibit 7.1 is a starting point for understanding SDAs' reactions to the study. It shows issues 
raised by those SDAs agreeing to participate and those declining to, listed in the order in which they were 
most commonly cited by all SDAs. Both participating and nonparticipating SDAs raised similar issues; 
the study sites were not an unusual group of local programs in which implementation of the study posed 
no problems. For example, 56 percent of participating SDAs expressed ethical and public relations 
concerns about how random assignment might be implemented in their program, while 62 percent of 
SDAs declining to participate raised these issues. Overall, the most commonly cited concerns involved 
(1) potential controversies surrounding the study because of ethical and public relations problems with 
random assignment or with the denial of services to controls, (2) the study's effect on meeting recruitment 
goals, and (3) the study's possible effect on performance standards. Eighty-eight percent of SDAs 
agreeing to participate in the study expressed at least one of these interrelated concerns, a figure slightly 
higher than the 83 percent for SDAs not participating (these are not shown in Exhibit 7.1). 

SDAs agreeing to participate were significantly more likely to raise a concern about possible 
effects on performance standards, potential grievances (e.g., by members of the control group), and 
problems recruiting out-of-school youths. Typically, these SDAs had more detailed discussions with the 
researchers, and they had to explore all aspects of the study to develop detailed plans for its 
implementation. 

Local circumstances could largely determine the importance of these issues to each SDA and the 
willingness and ability of local staff to work with the researchers to address them. Differences in 
circumstances among SDAs could include objective, easily measured characteristics — such as those 
reported in the JTPA Annual Status Report (JASR) - for which national comparisons are possible. But 



'^This increase in site payments was funded by a shift from a baseline survey for all sample members to a client- 
completed Background Information Form, which was reviewed by site staff. In light of the reliance on random 
assignment to create the treatment and control groups, the Department and the research team felt that the type of 
detailed baseline information that could be obtained only through a survey was not essential. 
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EXHIBIT 7.1 



PERCENT \GE OF SDAs CITING CONCERNS ABOUT THE STUDY, 
BY PARTICIPATING AND NONPARTICIPATING SDAs 

Participating Ncmparticipating 

SDAs SDAs AU SDAs 

Concera (%) (%) (%) 

Ethical and Public 
Relations Implications of: 
Random assignment 

in social programs S6.3 62.3 61.8 
Denial of services 

toamtrols 50.0 54.7 54.4 

Potential Negative E^ect of 
Creation of a Control Group 
on Achievement of Client 

Recruitment Goals 62.5 46.7 47.8 
Potential Negative Impact on ■ 

Performance Standards 68.8 22.2 25.4 *** 

Implementation of the Study 
When Service Providers Do 

hitake 18.8 21.2 21.1 
Objections of Service 

Providers to the Study 12.5 17.9 17.5 
Potential Staff Administrative 

Burden 18.8 16.0 16.2 
Possible Lack of Support by 

Elected Officials 6.3 16.5 15.8 
Legality of Random Assignment 

and Possible Grievances 37.5 12.7 14.5 ** 

Procedures for Providing 
Controls with Referrals 

to Other Services 25.0 13.2 14.0 
Special Recruitment Problems 

for Out-of-School Youths 31.3 9.0 10.5 ** 



Sample Size 16 212 228 



SOURCE: Calculations for this table are based on responses from 228 SDAs contacted 
about possible participation in the National JTPA Study. 

NOTES: Concerns noted by less than 5 percent of the SDAs are not listed. 

When totaled, the percentages in each column are over 100.0 percent because 
SDAs could raise more than one concern. 

A two-tailed t-test was applied to differences between participating and non- 
participating SDAs. Statistical significance levels are indicated as * ^ 10 percent; 
** = 5 percent; *** = 1 percent. 



they could also include less clear-cut, but still important, differences in leadership style or desire to be 
involved in the national policy debate. 

As the rest of this chapter shows, the 16 study SDAs are quite similar to all SDAs nationally on 
the characteristics included in the JASR data.^^ Therefore, an SDA's decision apparently was affected 
in important ways by other factors. The experience of the researchers suggests that such intangibles as 
the personality of the SDA director or PIC chairperson, local staffs perception of the recruiting and 
program intake challenges posed by random assignment and the creation of a control group, and local 
political circumstances in the SDA were all important influences. Also important was past favorable 
exposure of SDA officials to policy research, either through having done this in a previous job or through 
previous participation in a study .^^ 

There did appear to be a link between the administrative structure of an SDA and the likelihood 
of participation in the study. As discussed in Chapter 2, many different interests are represented within 
the JTPA structure: SDA managers, SDA line staff, service providers under contract to the SDA, members 
of the PIC who may include business leaders and individuals from other social service or employment 
agencies, and elected ofHcials. This often led to a sequential decisionmaking process, where these groups 
were contacted — usually in the order listed above — and there support sought. If those at any stage 
opposed the study, this could stop the deliberations. Experience during site recruitment suggested that 
those SDAs with a simpler administrative structure and fewer stages involved in deliberations over the 
study were more likely to participate. 

Site reaction may also have been influenced by stafTs impression about how "good" impacts 
would be in their site. However, this could only have been speculation, since prior to the study no SDA 
had any information on tfie impact of its program. The only available indicators of site success, the 
outcome-based JTPA performance standards, measured a concept of site performance very different from 
impacts. In fact, at the time the study began, there was much debate about whether SDAs that did well 
on the performance standards were doing so by recruiting people who would do relatively well on their 
own, even without the program. If this were true, "strong performers" could have poor rather than strong 



^^Scc also Doolittle and Traegcr, 1990, for a direct comparison of SDAs that agreed or declined to participate. 
The conclusions are similar to those presented in this chapter. 

^'For example, an SDA official in one site was a former Department employee, while those in another site had 
previously participated in another social experiment. 
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impacts. At any rate, as the later analysis in this chapter shows, participating sites are very similar to the 
national picture on JTPA performance. 

Furthermore, now that impacts have been calculated for the first 18 months of follow-up, it is 
possible to determine whether, in fact, there is any relationship - for the sites in the study - between 
success in meeting or exceeding the JTPA performance standards and estimated impacts. Using a variety 
of measures of program performance in force during the late 1980s, one finds no clear relationship 
between site performance and impacts. For example, there is virtually no relationship between how well 
a site did on the adult standards of "entered employment rate" and average wage at placement vis-a-vis 
site impacts for adults." Similar results were found on the relationship of youth performance measures 
(entered employment rate and positive termination rate) to site-level impacts for youths." 

m. The Characteristics of the 16 Study Sites 

This section reviews three features of study sites: the characteristics of the local population and 
economy, program duration and cost during the period of random assignment, and SDA success in 
meeting JTPA performance standards.^* On these measures, the study sites show considerable variation, 
but their average is similar to the national picture. 

A. Characteristics of the Local Population and Economy 

As discussed in Chapter 2, and depicted in Exhibits 2.1 and 2.2, the 16 study sites are spread 
throughout the nation and include a mix of types of locale. Exhibit 7.2 provides more evidence of the 
sites' diversity. Three SDAs made up solely of cities stand out in population density (column 1 of the 
table): Jersey City, Oakland, and Providence. Butte, Larimer County, Northwest Minnesota, and - to 
a lesser degree - Cedar Rapids and Springfield fall at the other extreme and are largely rural in character. 
Fort Wayne's relatively low population density is an average over eight counties that are predominately 



"Various pcrfomiancc measures were tested including: the percentage by which the site exceeded its applicable 
standard for adult entered employment rate in individual years, the average of this for the three years in which 
random assignment was in place, and the extent to which sites exceeded the applicable standard for average wage 
at placement. Various regression equations were estimated, and in no case was the coefficient on the performance 
measure even close to being different from zero at conventional standards of statistical significance. In addition, the 
R' for equations estimated in no case exceeded .04. 

^^In the estimation of impacts reported in the 18-month impact report, all observations arc weighted equally. 
In fiimre reports from the project, the implications of alternative weighing approaches will be explored. For example, 
the nonexperimcntal team is examining alternatives such as weighing sites by the "p" value of net impacts or using 
harmonic means weights to reduce the effects of extreme values for site impacts on the overall impact estimate. 

^^Sce Kemple, Doolittle, and Wallace, 1993, for a more complete discussion of the material in this section. 
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rural in character. The average population density for the sample exceeds that of SDAs as a whole, partly 
because rural SDAs with a small number of participants were not recruited to participate in the study.^^ 

The poverty rate, presented in column 2 of Exhibit 7.2, shows similar variety, although the sample 
and national average are quite close. Sites located in large, heavily minority metropolitan areas (Jersey 
City, Oakland) have the highest poverty rates, but other urban sites with minority populations such as 
Corpus Christ! (Hispanic) and Jackson (black) also have higher-than*average rates. Four other sites (Coosa 
Valley, Georgia; Heartland, Florida; Northwest Minnesota; and Springfield, Missouri) also have poverty 
rates slightly above the sample and national average. 

Economic characteristics at the time of random assignment (summarized in the remaining columns 
of Exhibit 7.2) reflect differences in regional economic conditions and the local economic base. On all 
these measures, the sample average is similar to the national average. As is the case nationally, the 
average unemployment rate among the sites masks great differences. The variation in SDAs' average 
earnings reflects higher average earnings in urban areas than in rural areas (e.g., Oakland versus Northwest 
Minnesota) and the importance of high-wage industries in some sites (e.g., petroleum in Corpus Christi 
and heavy manufacturing in Decatur).^^ 

B. Duration of Service and Program Cost 

Title II-A programs in the 16 sites varied greatly in average duration of enrollment and cost, but 
the averages for all study sites and all SDAs nationally are very similar, as shown in Exhibit 7.3. Program 
cost per adult terminee reflects length of enrollment, service emphasis in the SDA (discussed in Chapter 
3), and higher service costs (based on higher rent and salaries) in large metropolitan areas such as Jersey 
City and Oakland. These costs also vary because of differences in the ability of sites to draw on non- 
JTPA funding sources to secure services for JTPA participants. For example, some sites were more 
successful than others in placing clients in subsidized training programs (e.g., community colleges where 
the tuition does not cover all service expenses) or in "leveraging" funding from other agencies for parts 
of the service plan (e.g., securing assistance with child care or transportation from the welfare agency). 

C. Site Performance Standards 

JTPA performance standards, as noted in Chapter 1, assess program success in achieving a variety 



^^The averages for the sample and for the nation are unweighted. Thus, all sites, regardless of the number of 
individuals served, are treated equally in calculating both averages. 

Average earnings are calculated by dividing the total payroll reported to federal and state Unemployment 
Insurance programs by employers in the SDA by the number of employees in the SDA. 
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EXHIBIT 7.3 



SELECTED CHARACTERISTICS OF THE JTPA TITLE II-A PROGRAMS 
AT THE NATIONAL JTPA STUDY SITES 



Federal Program 





/werage iNumoer 


Average Number of Months Enro led 


ijosi r^er /\auic 




KJl i Cr QUXlCCb 




X \j\hjis> 


lemunec 










^>VlQft7^fiQ^ 
(^r I iyo/*^oyj 


DuujCj XVI oni. 










v^COAr ivvpiUSy iOWa 


ACQ 






2,ZlZ 


coosa vaiicy, vja. 




9 7n 


ca 
J.J** 


Z,46l 






7.85 


7.62 


Z,D /U 


Decatur, III. 


525 


6.77 


5.77 


3,039 


Fort Wflvne Ind 


1 1QS 


3.70 


7.16 


i,.>Oi 


Heartland, Fla. 


1,793 


3.54 


5.54 


1,782 


Jackson, Miss. 


1,227 


1.77 


3.46 


1,897 


Jersey City, N.J. 


853 


3.62 


3.23 


3,637 


Larimer County, Colo. 


354 


7.39 


6.00 


1,937 


Marion, Ohio 


714 


6.24 


6.08 


2,199 


Northwest Minnesota 


430 


6.62 


6.54 


2,371 


Oakland, Calif. 


1,396 


3.77 


3.93 


2,539 


Omaha, Neb. 


1,111 


2.46 


2.85 


2,404 


Providence, R.I. 


503 


1.62 


1.23 


2,841 


Springfield, Mo. 


938 


4.00 


3.93 


1,898 


Site Average 


899 


4.63 


4.80 


2,377 


National Average 


1,177 


4.57 


4.97 


2,241 



SOURCES: JTPA Annual Status Reports (JASR) for program years 1987-89. 



NOTE: Data are averages for all JTPA Title II-A terminees during program years 1987-89. 
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of program goals. Thus, it is important to ascertain whether the study sites are top performers under this 
system or, instead, include SDAs with the diversity of performance seen nationally. Exhibits 7,4 and 7.5 
show that the study sites do include such diversity and as a group performed about the same as the 
national average. 

These tables show data for program year 1988, the year in which the largest proportion of the 1 8- 
month study sample were randomly assigned. (This sample is discussed in the next section of this chapter.) 
For each of three post-program outcomes, the tables show (1) the actual performance of the site, (2) the 
performance that was "predicted" by the Department's regression model used by most states to adjust the 
national performance standards to reflect the characteristics of local terminees and labor market conditions, 
and (3) the difference between actual and predicted performance. Youth positive terminations include job 
placement at termination, achievement of "employment competencies," school completion, enlistment in 
the armed forces, and enrollment in other non-Title- II-A training programs. On all three measures, thie 
study sites include some that perform much better than their adjusted standard (i.e., those with large 
positive numbers in the difference column), some close to this standard, and some that fail to meet it (i.e., 
those with negative numbers). 

rv. A Comparison of the Research Sample and JTPA Participants Nationally 

This section summarizes an analysis of the background characteristics and JTPA activities of the 
individuals in the 1 8-month follow-up sample compared to those of all persons served by JTPA nationally 
during the period of the study .^^ It begins by defining the two samples being compared and then presents 
results of the analysis.^* 

A. The Study Research Sample 

The experimental sample of 20,601 is one of the largest ever produced in a social experiment. As 
Exhibit 7.6 shows, it includes very large subsamples for the four major subgroups (adult women, adult 
men, female youths, and male youths), ranging in size from 2,563 to 8,057 persons. The subsamples for 



"This section is based on Appendix B of the accompanying 18-month impact report. Bloom ct al., 1993, which 
contains a more detailed discussion of the methodology used in the comparisons and tables supporting the conclusions 
of this section. 

^*For a more complete discussion of the material in this section, based on an analysis of the full experimental 
sample and JTPA participants nationally, see Bloom, 1991. It reaches a conclusion similar to the one in this section 
on the similarity of the background characteristics of the two groups, but docs not analyze JTPA service receipt by 
the two groups. 
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EXHIBIT 7.4 



JTPA PROGRAM OUTCOMES FOR ADULTS, BY SITE 





Adult Entered Employment Rate 




Actual 


Predicted (a) 




Site 


(%) 


(%) 


Difference 


Butte, Mont. 


74.0 


67.1 


6.9 


Cedar Rapids, Iowa 


76.9 


73.1 


3.8 


Coosa Valley, Ga. 


83.5 


68.2 


15.3 


Corpus Christi, Tex. 


72.0 


67.1 


4.9 


Decatur, 111. 


79.4 


65.1 


14.3 


Fort Wayne, Ind. 


84.0 


72.4 


11.6 


Heartland, Fla. 


74.5 


68.7 


5.8 


Jacks<m, Miss. 


67.6 


69.2 


-1.6 


Jersey City, N.J. 


86.5 


64.2 


22.3 


i^nnier VA>un^, i.A>iO. 


68.0 


69.5 


-1.5 


Marion, Ohio 


55.5 


59.4 


-3.9 


Northwest Minnesota 


73.5 


69.1 


4.4 


Oakland, Calif. 


67.4 


66.1 


1.3 


Omaha, Neb. 


65.0 


65.7 


-0.7 


Providence, R.I. 


74.3 


70.2 


4.1 


Springfield, Mo. 


89.0 


76.4 


12.6 


Site Average 


74.5 


68.2 


6.2 


National Average 


74.2 


67.3 


6.9 



SOURCE: JTPA Annual Status Report (JASR) for program year 1988. 

NOTES: There may be slight discrepancies in the reported differences of the averages 
because of rounding, and there may be small adjustments to the rates that were 
made by states and not reported in JASR. 

(a) The predicted entered employment rate is based on the JTPA performance 
standard reported in JASR (PY1988), 
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EXHIBIT 7.6 

SIZE OF THE EXPERIMENTAL SAMPLE, 
BY TARGET GROdP AND SERVICE STRATEGY 



Target Group 


dlassroom 
Training 


OJT/JSA 


Services 


All 
nil 

Strategies 


Adult Men 


1,586 


3,182 


2,080 


6,848 


Adult Women 


3,409 


2,660 


1,988 


8,057 


Male Youths 


732 


8S6 


975 


2,563 


Female Youths 


1,362 


714 


1,057 


3,133 


All Target Groups 


7,089 


7,412 


6,100 


20,601 
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each of the three service strategies are also large, ranging in size from 6,100 to 7,412 persons. 

The 18-month follow-up survey was fielded for all 20,501 members of the experimental sample for 
whom contact information was available.^' Surveys were completed for 84 percent of the sample. Of 
the variables used to compute central employment and training outcomes for the impact analysis, item- 
specific response rates met or exceeded 90 percent of all respondents in all cases. 

As discussed in a later section, the survey was fielded in a way that produced varying lengths of 
follow-up, anywhere from 13 to 22 months. The methods available for analyzing variable-length follow- 
up are quite complicated.'^ Problems of analysis occur because the sample changes as the length of 
follow-up increases. Some of the individuals analyzed in data collected 13 months after random 
assignment would not be in the sample as follow-up lengthened. Even with complicated statistical 
modeling, it would be difficult to determine if changes in impacts over time occurred because of time 
trends in program impacts or differences in the composition of the sample. 

To simplify the analysis of program implementation and impacts, the researchers selected a sample 
of individuals for whom survey contact was scheduled at least 18-months after their date of random 
assignment. Also in the interest of simplifying the analysis, we excluded from the 18-month impact 
analysis a random sample of treatment group members who had been randomly assigned at a 
treatment/control group ratio different from the standard two-to-one ratio.'* This occurred in five SDAs 
that had special difficulties recruiting applicants; in these cases, the random assignment ratio was 
temporarily changed to a three-to-one or six-to-one ratio.'^ The resulting sample, called the "18-month 
study sample" in this report and the accompanying 18-month impact report, consists of 17,026 individuals 
assigned to either the treatment group or the control group before July 1989.'' 

For the 18-month study sample (shown in Exhibit 7.7), the survey response rate was 85 percent, 
with rates for each target group ranging from 80 percent for adult men to 83 percent for male youths to 
88 percent for aduh women and female youths. These response rates produced 14,442 respondents to the 
survey. 

For the comparison of the research sample with all JTPA participants, the analysis focuses on 



^'Contact infomiation was missing for 100 members of the sample. 

'°For an earlier analysis of data with variable length follow-up, sec Maynard, 1980. 

''This avoided the need to use a weighing schema in calculating impact estimates. 

"This exclusion of "extra" treatment group members resulted in a dropping of 473 persons from the 18-month 
study sample. 

"The final impact report, based on longer follow-up, will rely on both a second follow-up survey and 
administrative records and will analyze the experiences of a larger sample. 
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EXHIBIT 7.7 



ERIC 



SIZE OF THE 18-MONTH STUDY SAMPLE, 
BY TARGET GROUP AND SERVICE STRATEGY 



Target Group 


Classroom 
Training 


OJT/JSA 


Other 
Services 


All 

Strategies 


Adult Men 


1,091 


2,278 


1,150 


4,519 


Adult Women 


2,579 


2,074 


1,173 


5,826 


Male Youths 


536 


622 


615 


1,773 


Female Youths 


1,055 


551 


718 


2,324 


All Target Groups 


5,261 


5,525 


3,656 


14,442 



SOURCE: Adapted from Bloom et aL, 1993, which used unadjusted frequencies 
from Background Information Form responses. 
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individuals in the treatment group who enrolled in JTPA; this permits comparisons of JTPA service 
receipt.^^ Approximately 64 percent of all members of the treatment group v^ere enrolled in JTPA at 
some point during the 18 months following random assignment. 

E, The National Comparison Sample 

National data on JTPA enrollees are available from the U.S. Department of Labor's ongoing Job 
Training Quarterly Survey (JTQS), which contains data on individual JTPA participants collected from 
the administrative records of a nationally representative sample of 142 SDAs. In the analysis that follows, 
the 18«month follow-up sample is compared to representative national samples of program participants 
and program terminees who enrolled in JTPA Title II-A during the period of random assignment.^^ Both 
data sources contain information on age, ethnicity, education status, employment status, and public 
assistance receipt. 

C. A Comparison of the Two Groups 

Differences in background characteristics available in both data sources are small, as seen in 
Exhibits 7.8 through 7.1 1, and there is no consistent pattern of differences across the target groups. More 
specifically: 

• Almost all background characteristics for the target groups in the experimental 
sample are similar to their counterparts in the national comparison sample. 

• Few characteristics differ substantially for any single target group. 

• No characteristic differs substantially for all target groups. 

• Those characteristics that do differ for more than one target group do not do so in 
a consistent way. 

The one important difference to be noted is a lower level of educational attainment — i.e., receipt of a 



"As discussed in Chapter 5, the treamicnt group and control group are very similar on all measured 
characteristics. Within the treatment group, those who enrolled and those who did not are again very similar. For 
the details of this analysis, see Bloom et al., 1993. 

^'The national comparison enrollee sample includes 12,289 adults and out-of-school youths in the JTQS who 
enrolled in JTPA Title II-A while random assignment for the study was under way. However, the JTQS contains 
program data on program participation and status at temiination from the program only for JTPA temiinees. 
Therefore, the national comparison terminee sample includes 13,430 adults and out-of-school youths in the JTQS 
who enrolled during the period of random assignment and for whom termination information is available. 
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EXHIBIT 7.8 



SELECTED BASELINE CHARACTERISTICS OF ADULT FEMALE ENROLLEES 
IN THE 18-MONTH STUDY SAMPLE AND THE NATIONAL COMPARISON SAMPLE 



Characteristic 
and Subgroup 


JTPA Enrollees 
in the 
18-Month 
Study Sample 
(%) 


JTPA Enrollees 
in the National 
Comparison 
Sample 

(%) 


JTPA Terminees 
in the National 
damns ri^nn 
Sample 

(%) 


Age 








21 and under 


0.0 


0.0 


0.0 


22-29 


44.0 


42.6 


41 8 


30-44 


44.0 


41.5 


42.1 


45-54 


7,4 


8.1 


7.6 


55 and over 


4.6 


7.9 


8.5 


Ethnicity 








White, non-Hispanic 


56.2 


54.0 


53.9 


Black, non-Hispanic 


27.9 


32.0 


32.3 


Hispanic 


12.2 


11.0 


10.5 


Other 


3.6 


3.1 


3.3 


Education 








High school diploma or GED 


73.9 


70.9 


71.7 


No high school diploma or GED 


26,1 


29.1 


28.3 


Employment Status 








Employed 


15.0 


15.2 


14.9 


Not employed 


85.0 


84.8 


85.1 


Public Assistance Receipt 








Receiving any public assistance (a) 


54.6 


60.2 


61.4 


Receiving AFDC 


34.6 


40.4 


36.3 


Receiving Food Stamps 


49.6 


47.8 


50.1 


Sample Size 


2,895 


5,032 


5,395 



SOURCE: Adapted from Bloom et al., 1993, which used unadjusted frequencies from Background Informa- 
tion Form responses and the Job Traming Quarterly Survey (JTQS). 



NOTES: Data are for program years 1987-89. 

Distributions may not total 100.0 percent because of rounding. 

(a) "Any public assistance" includes the following: AFDC, Food Stamps, housing assistance, and 
other cash assistance. It does not include unemployment insurance. 
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EXHIBIT 7.9 



SELECTED BASELINE CHARACTERISTICS OF ADULT MALE ENROLLEES 
IN THE 18-MONTH STUDY SAMPLE AND THE NATIONAL COMPARISON SAMPLE 



Characteristic 
and Subgroup 


JTPA EnroUees 
in the 
1 S-Month 
Study Sample 
(%) 


JTPA Enrollees 
in the National 

Sample 

(%) 


JTPA Terminees 
in the National 

Sample 

(%) 


Age 








21 and under 


0.0 


0.0 


0.0 


22-29 




43.2 


4°) 4 


30-44 


42.5 


42.2 


40.9 


45-54 


8.0 


7.4 


7.7 


55 and over 


4.0 


7.2 


7.9 


Ethnicity 








White, non-Hispanic 


57.3 


58.4 


58.0 


Black, non-Hispanic 


27.6 


26.2 


27.1 


Hispanic 


10.3 


11.8 


11.4 


Other 


4.4 


3.6 


3.5 


Education 








High school diploma or GED 


65.7 


69.6 


67.8 


No high school diploma or GED 


34.3 


30.4 


32.2 


Fmnlovment Stfltuj: 








Employed 


12.7 


14.3 


13.6 


Not employed 


87.3 


84.8 


85.1 


Public Assistance Receipt 








Receiving any public assistance (a) 


31.3 


34.9 


34.8 


Receiving AFDC 


6.3 


10.1 




Receiving Food Stamps 


28.4 


25.8 


26.5 


Sample Size 


1,818 


3,835 


4,293 



SOURCE: Adapted from Bloom et al., 1993, which used unadjusted frequencies from Background Informa- 
tion Form responses and the Job Training Quarterly Survey (JTQS). 



NOTES: Data are for program years 1987-89. 

Distributions may not total 100.0 percent because of rounding. 

(a) "Any public assistance" includes the following: AFDC, Food Stamps, housing assistance, and 
other cash assistance. It does not include unemployment insurance. 
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SELECTED BASELINE CHARACTERISTICS OF FEMALE YOUTH ENROLLEES 
IN THE 18-MONTH STUDY SAMPLE AND THE NATIONAL COMPARISON SAMPLE 



Characteristic 
and Subgroup 


JTPA Enrollees 
in the 
18-Month 
Study Sample 
(%) 


JTPA Enrollees 
in the National 
Comparison 
Sample 
(%) 


in the National 
Comparison 
Sample 
(%) 


Ace 








16-19 


59.3 


61.0 


60.4 


20-21 


40.7 


39.0 


39.6 


22 and over 


0.0 


0.0 


0.0 


Ethnicity 








White, non-Hispanic 


51.2 


48.0 


48.2 


Black, non-Hispanic 


28.2 


37.2 


34.2 


Hispanic 


18.8 


11.7 


14 0 


Other 


1.8 


3.2 


2.8 


Education 








High school diplonoa or GED 


47.5 


59.1 


55.7 


No high school diploma or GED 


52.5 


40.9 


44.3 


Employment Status 








Employed 


15.6 


13.3 


12.1 


Not employed 


84.4 


86.7 


87.9 


Public Assistance Receipt 








Receiving any public assistance (a) 


46.7 


53.0 


54.0 


Receiving AFDC 


26.6 


27.1 


30.3 


Receiving Food Stamps 


39.6 


39.5 


42.1 


Sample Size 


1,200 


1,725 


1,920 



SOURCE: Adapted from Bloom et al., 1993, which used unadjusted frequencies from Background Informa- 
tion Form responses and the Job Training Quarterly Survey (JTQS). 



NOTES: Data are for program years 1987-89. 

Distributions may not total 100.0 percent because of rounding. 

(a) "Any public assistance" includes the following: AFDC, Food Stamps, housing assistance, and 
other cash assistance. It does not include unemployment insurance. 
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SELECTED BASELINE CHARACTERISTICS OF MALE YOUTH ENROLLEES 
IN THE 18-MONTH STUDY SAMPLE AND THE NATIONAL COMPARISON SAMPLE 



Characteristic 
and Subgroup 


JTPA Enrollees 
in the 
18-Month 
Study Sample 
(%) 


JTPA Enrollees 
in the National 
Comparison 
Sample 

(%) 


JTPA Terminees 
in the National 
Comparison 
Sample 

(%) 


A OP 








16-19 


63.9 


61.7 


63.2 


20-21 


36.1 


38.3 


36.8 


22 and over 


0.0 


0.0 


0.0 


Ethnicity 








White, non-Hispanic 


55.2 


55.7 


54.2 


Black, non-Hispanic 


26.5 


28.8 


27.7 


Hispanic 


15.8 


12.2 


15.8 


Other 


2.4 


3.3 


2.3 


Education 








High school diploma or GED 


38.3 


44.7 


45.5 


No high school diploma or GED 


61.7 


55.3 


54.5 










Employed 


12.1 


13.7 


13.2 


Not employed 


87.9 


86.3 


86.8 


Public Assistance Receipt 








Receiving any public assistance (a) 


30.0 


32.8 


32.1 


Receiving AFDC 


5.7 


8.3 


9.0 


Receiving Food Stamps 


26.0 


23.4 


25.2 


Sample Size 


976 


1,697 


1,826 



SOURCE: Adapted from Bloom et al., 1993, which used unadjusted frequencies from Background Informa- 
tion Form responses and the Job Training Quarterly Survey (JTQS). 



NOTES: Data are for program years 1987-89. 

Distributions may not total 100.0 percent because of rounding. 

(a) '^Any public assistance*" includes the following: AFDC, Food Stamps, housing assistance, and 
other cash assistance. It does not include unemployment insurance. 



high school diploma or its equivalent, a GED — among youths in the study sample than among youths 
in the national comparison sample.^^ 

It is also important to compare JTPA activities and status at program termination of the 1 8-month 
follow-up enroUee sample with those of the national comparison termi ice sample (for which program 
participation information is available). Such an analysis shows: 

• Similar lengths of JTPA participation for the 18-month and national sample for each 
target group. 

• Greater participation in classroom training in occupational skills and job search 
assistance among the 18-month follow-up sample than among the national 
comparison sample. This is true for each target group. 

• Less participation in cn-the-job training and "miscellaneous services*' (defined in 
Exhibit 4.4, noted) among the 18-month follow-up sample than among the national 
comparison sample. This is also true for each target group. 

• Somewhat higher "entered employment" rates at termination for the national 
comparison sample in each target group and a higher wage at placement for the 
national comparison group for aduh women. For adult men and youths, wage at 
placement was similar for the research and national comparison sample, as were the 
positive termination rates for male and female youths. 

Though there are differences in the overall service mix for the experimental sample and the national 
JTPA program, much of the analysis of program impacts focuses on subgroups recommended for specific 
service strategies, including OJT/JSA, classroom training, and "other services."" Because of this 
disaggregation, readers will be able to see the implications of these differences in service mix for overall 
program impacts in the study sites. 



^^erc was also a slightly lower percentage of adults over 55 years of age in the study sample than in the 
national comparison sample because in some sites these older workers were excluded from the study. 
"These service strategics arc discussed in more detail in Chapters 5 and 6 of this report. 
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APPENDIX 

PROFILES OF THE SITES IN THE NATIONAL JTPA STUDY 



This Appendix presents a brief profile of each of the 16 sites participating in the National JTPA Study. 
The profiles are in alphabetical order by the site names used in the study reports (with the formal SDA 
listed separately, if different).^ The profiles depict the SDAs and their local environments as of the time 
the sites were selected for the study and the random assignment process was going on (primarily program 
years 1986-88), and therefore do not reflect subsequent changes.^ 

A. Butte, Montana; Concentrated Employment Program 

1. Study Context . The Concentrated Employment Program (CEP) serves 10 counties in 
western Montana. The CEP SDA has approximately 125,000 residents, of whom about 37,000 live in 
Butte; 24,000 live in Helena. The remaining parts of the SDA are very rural, with no towns over 15,000 
in population. SDA residents are primarily white, and 7.5 percent of all families had incomes below the 
federal poverty level in 1979. 

The SDA's economy has traditionally been based on mining, timber, and ranching. In the late 
1970s and early 1980s, major layoffs in mining and related industries occurred ?n Butte and Anaconda, 
causing a decline in the local economy in these areas. The state capital in Helena provides white-collar 
employment, while Butte's economy is gradually diversifying. Unemployment in the SDA was 9 percent 
in 1984 and gradually declined to 7.4 percent in 1987 and 7 percent in 1988. The average annual wage 
for workers in the SDA was $16,700 in program year 1986. 

2. The Program . During the period of random assignment, the Montana Department of Labor 
and Industry, Employment Policy Division, was the grant recipient for the SDA, which has a Private 
Industry Council (PIC) appointed by the CEP Council of County Commissioners.^ The Council of 



^Programs not included in the sample, such as Titles III and II-B, and services to in-school youths through Title 
II-A are not specifically discussed. 

^To ensure a consistent base for comparison across sites, 1980 Census data were the primary source of 
information on population and size, while the JTPA Annual Status Report (JASR) was used for information on labor 
market conditions. For some variables, these profiles list the conditions at the start of the study and Chapter 4 
presents additional, more recent information. SDAs' Annual Job Training Plans, and observations and data collected 
by the researchers during the implementation of the study, were also drawn on in developing these profiles. 

^On July 1, 1990, the operator of this SDA changed from the Montana Department of Labor and Industry, a 
government entity, to Montana Job Training Partnership, Inc., a private nonprofit organization formed by the two 
PICs in Montana. 
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County Commissioners also is consulted on JTPA policy and at times meets jointly with the PIC. The 
JTPA program is operated by the Montana Job Service Division, under contract with the Employment 
Policy Division. Four Job Service offices provide JTPA services: In Helena and Butte, separate JTPA 
intake offices serve clients, while in Anaconda and Dillon, the Job Service provides both its usual job 
listing services and JTPA in a single office. Thus, especially in Anaconda and Dillon, it is quite likely 
that the control group were offered access to the usual Job Service job listings and other job-locating 
services. 

The Job Service staff offer job search assistance and OJT in all offices and provide classroom 
training by referrals to other agencies. In Helena, a basic skills brush-up course, followed by clerical 
training, was an important service, preparing people for employment in the capital area, while in much 
of the rest of the SDA, on-the-job training was the most common service. In the rural parts of the SDA, 
the nearest provider of classroom training in occupational skills was many miles away. 

B. Cedar Rapids. Iowa; East Central Iowa 

1. Study Context . The East Central Iowa SDA includes the cities of Cedar Rapids (population 
1 10,000) and Iowa City (population 50,000). The total population in the SDA is approximately 330,000, 
and outside the two main cities the SDA is very rural. The largest SDA office is located in Cedar Rapids, 
and each of the surrounding five counties (Benton, Iowa, Johnson, Jones, and Washington) has a smaller 
office. About 6 percent of all families in the SDA had incomes below the federal poverty level in 1979. 

The local labor market is closely linked to the fortunes of agriculture, through direct production 
and processing of food products. Within the SDA, the University of Iowa is also a major employer. 
Unemployment remained at about 6 percent from 1984 through 1986, then dropped to 4.3 percent in 1987 
and to 3.6 percent in 1988 with growth in the local economy. The average annual wage in the SDA was 
^proximately $17,200 in program year 1986. 

2. The Program . The SDA (a private, nonprofit, multi-jurisdictional agency) administers the 
JTPA program. Each of its six offices is responsible for taking applications, determining eligibility, and 
assessing applicants. Staff in the offices are typically experienced, with many having been with the 
agency since the CETA program. 

Client recruitment was not a problem in this SDA, except for occasional problems enrolling 
youths. This was partly because, when the study began, many enrollees were carried into a new program 
year. In this second program year, funding was not as great as expected, so the target number of new 
enrollments dropped sharply, and the research sample in this site fell well below the expected size. 
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These funding problems had implications for the services provided in the SDA. In past years, 
intensive services such as on-the-job training (OJT) had been the major services offered. With the decline 
in funding, SDA managers encouraged staff to provide less intensive services to allow more applicants 
to be served. Nevertheless, staff continued to recommend intensive services such as OJT and classroom 
training for most applicants. 

C* Coosa Valtev^ Georgia 

1. Study Context The Coosa Valley SDA provides JTPA services for a 10-county area in the 
northwestern part of Georgia. The SDA is large and rural, and public transportation is nonexistent except 
in Rome, which is the largest city, with a population of 30,000. Services are provided through a 
decentralized structure in order to reach the total SDA population of 355,000, which is predominantly 
(more than 90 percent) white. Approximately 1 1 percent of the families had incomes below the federal 
poverty level in 1979. 

The most important employment sector is manufacturing, which employed 38 percent of the 
workforce in 1980. The textile and apparel industries dominant this sector. The service sector also 
supplies a significant number of jobs, employing 19 percent of the workforce. The unemployment rate 
started to decline just as random assignment began - from 7 percent in 1985 to 5.7 percent in 1986. By 
1988, however, the unemployment rate had climbed back up to 6.3 percent. The average wage in the area 
was $15,300 per year in program year 1986. 

2. The Program . A state-created but locally administered Regional Development Center 
(RDC) is the planning and administrative entity and grant recipient fttr the SDA. RDCs fulfill multiple 
labor-market-related functions in Georgia and receive fiinds from the state as well as from the programs 
they oversee, such as JTPA. The Coosa Valley RDC provides fiscal and program oversight and contracts 
out all services. The PIC, whose members include a number of the JTPA contractors, provides input to 
the RDC. The RDC contracted with Berry College to provide intake services as well as occupational 
training. Because of the large area served by the SDA, intake was conducted at provider sites and in 
central public locations within the counties (e.g., courthouses), as well as at Berry College's JTPA office. 
Following the determination of eligibility for services. Berry College staff made referrals to the 18 JTPA 
contractors that dealt with the population covered by the study. 

Community colleges, vocational colleges, technical schools, and private for-profit and not-for- 
profit organizations provided classroom training in a wide range of occupations. Community-based 

A-3 



organizations worked primarily with out-of-school youths in GED preparation, job search activities, and 
the development of youth competencies. The private contractors also arranged OJTs. 

D. Corpus Christi. Texas; Corpus Christi/Nueces County 

1. Study Context . The Corpus Christi/Nueces County SDA (previously called the Corpus 
Christi Job Training Program) is located in southeastern Texas on the Gulf of Mexico. Corpus Christi, 
with a population of 232,000, has 86 percent of the 270,000 county residents. Hispanics and whites have 
been represented almost equally in the population, at 47 percent each, but the Hispanic population is 
increasing. Black residents represent about 5 percent. Approximately 13 percent of the families in the 
SDA had incomes below the federal poverty level in 1979. 

The local economy, with many jobs tied to the oil industry as well as tourism, was affected by 
the major recession in Texas in the mid-1980s. The unemployment rate fluctuated from 9.1 percent in 
1984 to 12 percent in 1987 to 10.4 percent in 1988. The average annual wage for the area in program 
year 1986 was $18,500. 

2. The Program . At the beginning of program year 1988, the PIC was incorporated and 
assumed responsibility as grant recipient and administrative entity. Staff then became employees of the 
PIC. There are two JTPA offices in the SDA, although the site outside the city is not a fiiU-service center. 
The SDA is responsible for Title II-A programs, including 3 percent and 8 percent programs, as well as 
Titles II-B and HI. 

On-the-job training (OJT) and classroom training in occupational skills are the most frequently 
emphasized services, with some GED and job search services also provided. Except for assessment and 
intake, the PIC uses performance-based contracts and subcontracts all services to outside organizations. 
The primary classroom training contractor, a community college, provides classroom training in a wide 
range of occupational areas; JTPA participants are mainstreamed with other students. A GED class, 
however, is offered only to JTPA clients. The Texas Employment Commission, the public Employment 
Service, was responsible for administering the OJT program during most of the study period, although in 
the first year of the study, a private for-profit contractor fulfilled this function. Two community-based 
organizations were also major contractors to the SDA: LULAC conducts job search, job club, and job 
placement, originally as a stand-alone activity, but more recently as a follow-up component for those who 
have completed classroom training in occupational skills at the community college; SER provides 
concurrent GED and classroom skills training for out-of-school youths. The SDA expanded its own 
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■ services to include some job search activities. Cost-reimbursement contracts were used for individual 
referrals to specific vocational training courses, offered primarily through proprietary schools. 

■ E. Decatun Hlinois: Macon/De Witt Counties 

■ L Study Context , The Macon/De Witt Counties SDA, located in centml Illinois east of 
~ Springfield, serves the two counties. They have a total population of nearly 150,000. Decatur, a city of 

■ approximately 90,000 located in Macon County, is the home of the larger JTPA office; a second office 

■ is located in De Witt County in the town of Clinton, which has about 8,000 residents. Outside of Decatur, 
§m the SDA is primarily rural, with scattered small towns. Decatur has a significant black population, but 

■ the remaining parts of the SDA are primarily white. In 1979, 7.8 percent of the families in the SDA had 
^ incomes below the federal poverty level. 

■ The Macon/De Witt labor market continued to experience high unemployment throughout the 
^ 1980s. The unemployment rate was 10.7 percent in 1984, 10.2 percent in 1985, and 1 1.3 percent in 1986; 
H it declined slightly to 10.6 percent in 1987 and to 9.3 percent in 1988. Plant closings or layoffs in several 

large heavy manufacturing firms occurred in the 1980s, and these cuts caused a leveling off in wholesale 
H trade, fmance and insurance, and real e'^tate employment. Construction of a nuclear power plant in 

Clinton did increase the demand for labor in that part of the SDA. The average annual wage in the SDA 

■ was $22,300 in program year 1986. 

2. The Program . The JTPA grant recipient is Macon County, which administers the program. 

■ An experienced staff operates the program, with the director and assistant director having served in the 
agency since the CETA program. The two JTPA offices in Decatur and Clinton conduct their own intake, 

■ assessment, and service delivery. In light of the high unemployment rate in the SDA, staff have not 
needed to actively recruit clients; 40 percent of applicants have been walk-ins, and the bulk of the 

■ remainder have been referrals from public aid or the Employment Service. 

The primary services offered by the SDA are on -the-job training (OJT) and job search assistance, 

■ though classroom training in occupational skills and basic education were planned to increase in size 
during the period of the study. At the time the study started, a local community college operated the OJT 

■ and job search assistance activities. These were provided under an arrangement by which the SDA paid 
a specified portion of the salaries of college staff, with the payment unrelated to the number of people 

B served, though the college received bonuses when it placed certain groups in jobs. The SDA also funds 

basic education (through a regional school district) and work experience, with a combination of these two 

■ activities sometimes serving as a lead into an OJT. Part-way into the study period, the SDA became part 
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of a state economic development initiative, with responsibility for providing a large number of highly 
screened job applicants for a new factory. Because of the this, the SDA ended random assignment earlier 
than expected and the research sample in this SDA reached only two-thirds of the expected size. 

F. Fort Wavne> Indiana: Northeast Indiana 

1. Study Context . The Fort Wayne Area Job Training and Development Corporation or 
JobWorks, located in Indiana's northeastern comer, serves eight counties: Adams, Allen, De Kalb, 
Huntington, Noble, Steuben, Wells, and Whitley. Fort Wayne, a city of approximately 175,000 in Allen 
County, is the largest urban center in the SDA and the site of the SDA's central office. Outside of Allen 
County, the SDA is primarily rural, with many small towns. The SDA's total population is approximately 
500,000, about 93 percent of whom are white, with black residents making up the bulk of the remaining 
population. An estimated 6 percent of the families in the Fort Wayne area had incomes below the federal 
poverty level in 1979. 

A strong recovery from the recession of the early 1980s dropped the unemployment rate from 8.9 
percent in 1984, to 5.1 percent in 1987, to 4.8 percent in 1988. Some of the rural counties had 
unemployment rates well below this level. During 1988 and 1989, employers were having difficulty 
filling jobs, and some were seeking workers from outside the area. Manufacturing remains the largest 
employer in the SDA, providing jobs for one-third of all woricers in 1987. Between 1983 and mid-1987, 
manufacturing employment increased by 26 percent, or 16,500 jobs, though many of the new 
manufacturing jobs paid less than heavy industry jobs lost in the late 1970s and early 1980s. The average 
wage in the Fort Wayne area was $17,800 in program year 1986. 

2. The Program . The PIC is an incorporated nonprofit organization, which has an agreement 
with JobWorics to administer the JTPA program through an office in each of the eight counties in the 
SDA. Each JobWorks office is responsible for its own intake, assessment, service (delivery, and - during 
the period of the study - random assignment. A state initiative to merge the activities of the Indiana 
Employment Security Division (the state Employment Service) and JTPA led to greater cooperation 
between these two agencies at the local level and co-location of offices in many of the rural counties 
surrounding Fort Wayne. Even before this, most JobWorks applicants had already registered with the 
Employment Security Division, and those who had not were required to register as part of JTPA intake. 
Thus, it is very likely that individuals randomly assigned to the control group did receive the usual Job 
Service services. 
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Job Works offers a wide range of JTPA services, with the program emphasis varying by local 
office depending on the availability of service providers. The primary services offered are on-the-job 
training (OJT), job search assistance, and — in the areas close to Fort Wayne — classroom training in 
occupational skills. Because of the strong demand for employees in this SDA during much of the study, 
job search assistance was provided as a sole service to a relatively large portion of the sample in this site. 
OJT and job search assistance are operated by the JobWorks staff, while all classroom training in 
occupational skills programs are run by service providers, most of whom operate under performance-based 
contracts. The site subcontracts with about 20 service providers, with the usual number of clients rangin/^, 
from 5 to 40 per program. Service providers include local technical institutes and schools, community- 
based organizations, and other organizations that offer occupational training in a wide variety of areas or 
work with specific populations such as youths or handicapped individuals. 

G. Heartland, Florida 

1. Study Context . The Heartland SDA serves five counties (De Soto, Hardee, Highlands, Polk, 
and Okeechobee) in central Florida between Tampa and Orlando. Seventy-five percent of the SDA's 
population resides in Polk County, the site of Lakeland (population 50,000), the largest city in the SDA. 
Winteriiaven, a retirement community, is also in Polk County. The remaining four counties are much less 
developed, though they are dotted with a number of cities and towns. Eleven percent of families living 
in the SDA had incomes below the federal poverty level in 1979.. 

Employment in the SDA is a mixture of manufacturing, services, and (near Oriando) tourism. 
The unemployment rate in the SDA stood at 1 1.3 percent in program year 1984, declining gradually to 
9.4 percent in 1987 and to 8.1 percent in 1988. During the early 1980s, a major aircraft manufacturing 
facility closed, but after the start of the study, retail trade employment rose sharply with the opening of 
a major regional shopping center. Average annual wages in program year 1986 were $15,200, among the 
lowest of the SDAs in the study. 

2. The Program . The Heartland Private Industry Council (PIC), a nonprofit organization, 
operates the JTPA program within the SDA. At the time the study began, clients were served througli 
seven offices, three of which were in Polk County. On-the-job training positions (OJTs) are arranged by 
PIC staff, but, in a typical year, classroom training in occupational skills is subcontracted to approximately 
10 service providers, of which the Polk County Board of Education is the largest. Though these service 
providers do play the lead role in the recruitment of clients, the PIC staff conduct the JTPA intake and 



eligibility determination. The Florida Employment Security Commission (ESC) offices in Lakeland and 
Winterhaven also provide job search assistance under a contract with the PIC and do conduct JTPA intake. 

Normally, OJT and classroom training were the primary services for males, both adults and 
youth. However, in the study sample, job search assistance as a sole service was the most common 
activity for these groups. This unusual result occurred because of the small sample in this SDA and 
because of major staff changes in the SDA, which disrupted normal service patterns temporarily. For 
women, classroom training in occupational skills was the most common service, while fcr female youths 
miscellaneous non-training services were also important. While recruitment had not been a problem in 
the mid-1980s, finding appropriate JTPA-eligible applicants for the more demanding courses could be 
difficult. Soon after beginning random assignment, the PIC dismissed the senior management team of the 
SDA for reasons primarily unrelated to the study, and the SDA ended random assignment far short of its 
expected sample. 

H. Jackson. Mississippi; Capital Area 

I. Study Context. Three cities in northcentral Mississippi are served by the Capital Area SDA. 
They are Jackson, Clinton, and Pearl, with Jackson having 85 percent of the total population of 239,000. 
Tliirty-eight percent of the SDA*s population is black and other minorities. An estimated 13 percent of 
the families had incomes below the federal poverty level in 1979. 

The unemployment rate was 6.9 percent in 1985 and 6.7 percent in 1986, and declined to 5.6 
percent in 1988. The largest source of jobs is the service sector, with the state capital in Jackson 
providing white-collar employment. Oth^^r service industries, such as hospitals, are another major source 
of employment. The average wage in the SDA was $17,200 in program year 1986. 

2. The Program . The SDA is a consortium, with the city of Jackson serving as the 
administrative and planning entity and grant recipient for the program. Staff, employed by the city in the 
Department of Human and Cultural Services, are housed in one central office. The SDA is the grant 
recipient for Title II-A and II-B funds, but does not operate Title II-A 3 and 8 percent programs or Title 
III. 

The SDA emphasizes classroom training in both basic education (for youths) and a variety of 
occupational areas (for adults). Performance-based contracts are used to fund training in specific 
vocational areas such as health occupations, auto mechanics, truck driving, cooking, and security. JTPA 
is the primary source of business for a number of these contractors. Other contractors, including a 
community college and Jackson State University, arc funded to provide other services, including a 6- to 
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8-week job club/employability development program, which can lead to an on-the-job training position 
(an OJT), and adult basic education and GED (high school equivalency) classes. A total of 12 contractors 
were utilized during the study period. In addition, SDA staff develop OJT positions and place participants 
in employment. 

I. Jersey City, New Jersey; Corporation for Employment and Training^ Inc, 

1. Study Context . The Corporation for Employment and Training provides JTPA services to 
a single city area. This urban area, with a population of 224,000, is approximately 50 percent black, with 
the remainder of the residents divided primarily among whites, Hispanics, and Middle Eastern and Asian 
immigrants. With a population density of 7,000 per square mile and 14.7 percent of the families with 
incomes below the federal poverty level in 1979, the area shares problems similar to those confronting 
other large urban areas. 

The overall average wage during program year 1986 was $20,400. This average masks great 
differences in income within the community, with many longtime minority residents earning much less 
and many recent arrivals in "gentrified" neighborhoods working in high-paid jobs elsewhere in the 
metropolitan area. The unemployment rate dropped from 10.9 percent in 1984 to 7.9 percent in 1987 and 
then to 7.2 percent in 1988. However, many residents have sought employment outside the SDA because 
of a severe decline in manufacturing and the closing of many factories. The service sector, including 
financial institutions and distribution centers, are major employers, but wage rates are lower than in 
neighboring New York City. 

2. The Program . The city contracted with the Corporation for Employment and Training, a 
private, nonprofit organization that previously operated the CETA program, to administer the JTPA 
program.^ The corporation administered the Title II-A 3 percent and 8 percent programs and Title III 
in addition to the 78 percent and 6 percent funds, which are passed through to SDAs by the state. 

The on-the-job training (OJT) program was implemented by in-house staff, but compared to other 
SDAs in the study, this service was not emphasized. In-house staff also provided direct placements and 
job search. The corporation subcontracted for classroom training in occupational skills, using a total of 
23 providers throughout the study period. Contracts were performance-based and often directed to specific 
service occupations. Training was provided in such areas as clerical, financial, health and food services, 
and drafting occupations. One provider received funding to provide placement services to the Hispanic 



*Thc Corporation for Employment and Training lost its contract with Jersey City at the end of June 1990. 
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population during part of the study. Proprietary and business schools as well as vocational technical 
centers were primary contractors. The SDA increased its emphasis on individual referrals and contracts 
toward the end of the study implementation period. 

J. Larimer County. Colorado 

1. Study Context . Located in the northcentral part of the state, approximately 65 miles north 
of Denver, this rural county had a total population of 150,000 in 1980, 65 percent of whom lived in Fort 
Collins or Loveland, site of the two JTPA offices. By the mid-1980s, the SDA's population was 
^proximately 200,000 The population is predominantly white (92 percent in 1980); the largest minority 
is Hispanic (6 percent). Only 5.9 percent of the families were classified as having incomes below the 
federal poverty level in 1979. 

The county experienced strong growth in the late 1970s and early 1980s, but in the years 
immediately preceding the study, unemployment began to rise — from 4.8 percent in 1984 to 7.4 percent 
in 1987. In 1988, however, unemployment had receded to 6.6 percent. The service sector employs the 
largest number of workers, followed by trade and manufacturing. Colorado State University is the largest 
employer, with Hewlett Packard and Anheuser-Busch next in number of employees. The overall average 
wage in Larimer County was $17,500 in program year 1986, 

2. The Program . The county government is the planning and administrative entity and grant 
recipient for JTPA. SDA staffs are county employees in a department called Larimer County Employment 
and Training Services (ETS), but an active PIC guides the program. The SDA also administers the Title 
II-A 3 percent program and Title III. Fort Collins is the administrative headquarters and serves about 60 
percent of the SDA*s clients, with the Loveland office serving the remainder. When the study started, 
the SDA and PIC also oversaw the local Employment Service activities and staff', which are co-located 
with the SDA offices. But actions at the state level gradually resulted in a return to the more traditional, 
separate administrative structure. 

ETS, because of its relatively small Title II-A 78 percent allocation, serves the majority of its 
clients through short-term, low-cost, but highly structured activities, many conducted by its own personnel: 
Job club, a three-week program with its own training area and phone bank, is offered monthly in each 
office; a one-week assessment program and youth competency workshops are also ongoing. A workshop 
to teach job retention skills was added. 

Other programs are contracted to schools on a cost-reimbursement basis. Individuals 
recommended for the classroom training service strategy begin their time in training with a two-week 
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career planning program for JTPA participants conducted at Front Range Community College. ETS staff 
contribute to curriculum development and consult with instructors and clients in these programs. The high 
level of interaction between staff and other agencies is also evidenced by a formal network of 19 
community organizations, including groups not involved with JTPA, who meet regularly to foster 
coordination and develop solutions for individual problem cases. The SDA has received national 
recognition for its networking system. 

K. Marion, Ohio; Crawford/Hancock/MarionAVvandot Counties 

1. Study Context . Located in northcentral Ohio, this four-county SDA has a population of 
206,000. Marion is the largest city, followed by Findlay; each has a population of about 36,000. The 
population is predominantly white. Approximately 7 percent of the families had incomes below the 
federal poverty level in 1979. 

Economically, there is considerable variation within the area served by the SDA. The county of 
Hancock has had the lowest unemployment rate in Ohio, while the other counties have been adversely 
affected by declines in manufacturing. Overall, the unemployment rate for the area, which averaged 10.2 
percent between 1984 and 1986, dropped to 8.3 percent in 1987 and to 6.8 percent in 1988. The average 
annual wage in program year 1986 was $18,100. 

2. The Program . The PIC is incorporated and serves as the administrative entity and grant 
recipient. lis staff includes employment advisors, testers, and basic education instructors. The central 
administrative office is in Marion. Five JTPA training centers are spread throughout the SDA and are 
staffed by PIC personnel, who conduct extensive assessments and provide case management services. 
Through the training centers, participants can receive basic education and GED courses, using computer- 
assisted instruction, and basic education was a much more important service in this SDA than in most 
others in the study. PIC staff also conduct job club and arrange work experience and on-the-job training 
(OJT) placements. Classroom training in occupational skills is arranged on an individual basis through 
reimbursement contracts with vocational schools and technical colleges. 

Participants formeriy averaged approximately 40 weeks in the program as they moved through 
a variety of components designed to upgrade their skills. However, the decrease in the unemployment 
rate, relative to other SDAs in the state, resulted in a reduction in Title II-A funds, which led to less 
intensive service plans for clients. 
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L. Northwest Minnesota 

1. Study Context . The Northwest Minnesota Private Industry Council (PIC) serves seven 
counties (Kittson, Marshall, Norman, Pennington, Polk, Red Lake, and Roseau), which comprise a very 
rural area in the comer of the state abutting North Dakota and Canada. The population of the entire SDA 
is about 100,000, and its three largest towns (Crookston, East Grand Forks, and Thief River Falls) each 
has slightly under 10,000 residents. The population is primarily white, and about 11 percent of all 
families had incomes under the federal poverty level in 1979. 

The local economy is agriculture-based, with farming, food processing, and related trade being 
the major industries. The severe winters make for a short growing season, and sugar beets, potatoes, and 
wheat are the primary crops. The unemployment rate in the SDA stood at 10.3 percent in 1984 and 
i^ained above 9 percent until 1987, when it dropped to 8.6 percent. By 1988 unemployment had fallen 
to 8 percent. The average annual wage of $13,600 in program year 1986 was the lowest of any SDA in 
the study. 

2. The Program . The PIC is an incorporated nonprofit organization that is the JTPA grant 
recipient. It does some client recruitment, but the high unemployment rate has allowed the JTPA program 
to rely primarily on walk-ins and referrals from other social service ar.a education agencies. The 
Minnesota Job Service operates most of the JTPA program in the SDA under a subcontract from the PIC. 
In offices in Crookston and Thief River Falls, Job Service staff offer the usual state Employment Service 
job listings plus JTPA and other state-funded programs for welfare recipients. Thus, it is very likely that 
control group members received the usual Job Service services. The local community action agency 
conducts JTPA intake for a small older workers' program, which provides v/ork experience and job search 
assistance. 

In this SDA, on-the-job training (OJT) is emphasized more than in any other in the study; it is 
the most important service for both youths and adults. Job Service staff directly provide job search 
assistance, career exploration, OJT, and work experience. They refer people interested in classroom 
training in occupational skills to the local community college, technical institute, or campus of the state 
university. Those seeking adult basic education and a GED are referred to a local adult learning center 
for services. A local technical institute also provides an intensive job search assistance course, primarily 
for those participating in classroom training in occupational skills. 
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M* Oakland^ California 

!• Study Context The City of Oakland, with a population of 340,000, is one of 49 service 
delivery areas in California. Its population is approximately 40 percent white, 30 percent black, 25 
percent Asian, and 5 percent other minorities. There are 6,300 residents per square mile, and 8.7 percent 
of the families had incomes below the federal poverty level in 1979. 

Major industries are transportation (including shipping), manufacturing, government, and health 
care. Large employers are Kaiser, Pacific Bell, Merritt-Peralta Medical Supplies, Civic Gas and Electric, 
and Childrens Hospital. The unemployment rate fluctuated in the years preceding the study's 
implementation: It was 7.0 percent in 1984, 6.8 percent in 1985, 8.7 percent in 1986, 7.7 percent in 1987, 
and 6.6 percent in 1988. The overall average annual wage in the city was $22,000 in program year 1986. 

2. The Program . The PIC is incorporated and shares administrative responsibility for the 
program with the City of Oakland. The city is also the grant recipient and planning entity for the 
program. The PIC and city have defined their oversight and administrative functions so that the city is 
responsible for determining applicants' eligibility for programs and for tracking participation in the SDA's 
programs, while the PIC awards and monitors contracts for services. 

A subsidiary of the PIC - the Oakland Private Sector Corporation (OPSC) - is the largest 
contractor providing classroom training in occupational skills, on-the-job training (OJT), and other services 
10 all JTPA-eligibles and conducting an older woikers' program with Title II-A 3 percent and 78 percent 
funds. The Center for Employment and Training (CET), the second largest contractor, conducts training 
programs in four occupational areas — office occupations, shipping and receiving, auto mechanics, and 
maintenance - and provides a small number of OJTs The SDA also contracts with other outside 
agencies to provide services to the eligible population, though these contracts tend to be targeted to serve 
particular subgroups. All subcontracts are performance-based. Classroom training in occupational skills 
is the most common service overall for the adults included in the study, though for men job search as a 
sole service was slightly more common. 

fi. Omaha. Nebraska; Greater Omaha 

1. Study Context . Located in eastern Nebraska, this SDA serves approximately 500,000 people 
and includes the greater Omaha metropolitan area. The largest county is Douglas, with a population of 
398,000, of whom 312,000 live within the Omaha city limits. The SDA also includes Sarpy (population 
86,000) and Washington (population 16,000) counties. The majority of the population is white, with black 
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residents making up the largest minority group. In 1979, 6.6 percent of all families had incomes below 
the federal poverty level. 

The employment base includes large manufacturing and transportation employers as well as jobs 
in clerical and administrative work and sales and sales-related employment. The Strategic Air Command 
(S.A.C.) is the largest employer, followed by other large service-sector firms. 

The unemployment rate ranged between 4.7 percent and 5.6 percent in the four years preceding 
the study. In 1987, it was 5 percent, dropping to 4.4 percent in 1988. The average annual wage was 
$17,700 in program year 1986. 

2. The Program . The Greater Omaha SDA (formeriy called the Omaha Office of Employment 
Resources) provides JTPA services in the three-county area. The city of Omaha is the administrative 
entity and grant recipient, but an active PIC guides the program and is closely involved in major decisions. 
The SDA also administers Title III. Eight percent funds are used to enroll people in basic education 
classes combined with work experience; these are usually followed by on-the-job training (OJT) or 
classroom training in occupational skills. Eight percent funds are also used for classroom training in 
occupational skills and other classes at the local community college. 

Classroom training in occupational skills is the predominant service for Title II-A JTPA 
participants. During implementation of the study, most participants in this service were trained through 
five major contractors, predominantly community-based organizations. Classesprovided training in service 
occupations, such as health services, cooking, finance, office equipment repair, and clerical work. 
Contracts were performance-based. Job search is conducted by in-house staff, who also monitor and 
arrange OJT placements. 

O* Providence^ Rhode Island: Providence/Cranston 

1. Study Context The Providence/Cranston SDA serves these two adjacent, older urban areas 
in northeast Rhode Island. Providence, with a population of approximately 155,000, is the state capital 
and largest city in the state, while Cranston has approximately 70,000 residents. About 8 percent of the 
residents of the SDA are blacks, and all minority groups comprise 13 percent of the total population. Nine 
percent of the families in the SDA had incomes below the poverty level in 1979. 

The Providence/Cranston area has historically been one of the most industrialized in the country, 
with a higher than usual percentage of employment in manufacturing. In recent years, several of the 
state's major employers (makers of jewelry and silverware and the U.S. Navy) cut back employment, but 
other sectors (including services) grew with the New England recovery of the early and mid-1980s. 

A-14 



Unemployment in the SDA stood at 7.2 percent in 1984, but dropped to 4.3 percent in 1987 and to 3.7 
percent in 1988 as the SDA benefited from the then-booming regional high-tech and services economy. 
The average annual wage in the SDA in program year 1986 was $17,000. 

2. The Program . The Providence/Cranston Job Training Partnership (PCJTP) is the grant 
recipient and administers the program through offices in each of the two cities in the SDA. Recruitment 
of clients, done by both the PCJTP and service providers, has become increasingly a challenge as the 
area's unemployment rate dropped throughout the mid-1980s. Intake, eligibility determination, assessment, 
and counseling are performed by the SDA staff in each office, though the Cranston office must file case 
documentation of eligibility in the main Providence office before assessment can be scheduled. 

PCJTP staff arrange on-the-job training positions (OJTs) for clients, but the remainder of SDA 
services are provided by subcontractors operating under a mix of performance-based and cost- 
reimbursement contracts. Among the services provided in this way have been adult basic education, 
English as a Second Language, classroom training in occupational skills, vocational exploration programs 
(pre-employment skills and work experience for youths), and job clubs. 

P. Springfield^ Missouri: Job Council of the Ozarks 

1. Study Context . This SDA, located in southwestern Missouri, serves seven counties: 
Christian, Dallas, Greene, Polk, Stone, Taney, and Webster. Springfield (population 133,000), in Greene 
County, is the largest city and the location of the SDA's central oflFice. The SDA is primarily rural, with 
a total population of 304,000, of whom more than 90 percent are white. An estimated 10 percent of the 
families had incomes below the federal poverty level in 1979. 

The economy has been strong, with employment increasing in both the service sector, which 
employs the majority of the labor force, and manufacturing. Greene County also has had a strong 
agribusiness base. The unemployment rate has gradually declined in recent years, from 7.7 percent in 
1984, to 5.7 percent in 1987, to 5.3 percent in 1988; in areas that are strongly influenced by tourism, 
seasonal changes can bring the rate as high as 20 percent. The average wage in program year 1986 was 
$15,200. 

2. The Program . The City of Springfield Human Resources Department is the administrative 
entity and grant recipient. JTPA services are provided through the Job Council of the Ozarks, with full- 
service offices in Springfield, Branson, and Bolivar, a regularly staffed outreach office in Buffalo, and 
application sites in Kimberiing City and Marshfield. Staff in each office are responsible for intake. 



A.15 



ERIC 



170 



assessment, service delivery, and implementation of the study procedures. The PIC and local elected 
officials act as policymakers and planners and provide program oversight. 

Most of the classroom training in occupational skills programs are located in the Springfield area, 
and transportation is a problem for many SDA residents, so there is heavy reliance on on-the-job training 
(OJT) programs to meet the population's training needs, especially in the more rural parts of the SDA. 
Classroom training in occupational skills was provided primarily in health occupations, with programs such 
as licensed practical nursing and respiratory therapy. Welding, office occupations, and auto mechanics 
were also among the offerings. GED training was provided to dropout youths through a reimbursement 
contract with a community-based organization. Youths also received preemployment skills training and 
job placement services. Woric experience and a job-seekers' clinic were provided for both adults and 
youths. 
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